refine: refine interface of ManifestWriter

[ZENOTME]

This PR refine the write interface of ManifestWriter according to ManifestWriter from pyiceberg. It add 3 interface add, delete, existing which will rewrite some metadata of manifest entry, e.g. snapshot id, sequence number, file sequence number.

These refined interfaces are benefit for MergeAppend.( I' m working on it now.

[ZENOTME]

cc @liurenjie1024 @Xuanwo @Fokko @c-thiel

[liurenjie1024]

Thanks @ZENOTME for this pr!

[liurenjie1024]

It's kind of weird of manipulating arguments, how about make the arguments DataFile?

[liurenjie1024]

Applies to other apis.

[ZENOTME]

I think the reason here use ManifesEntry is that in some case we will add entry from other Manifest. In this case, there are some info we need from original ManifsetEntry. E.g. when we add the delete manifest entry, we change the snapshot id and keep the original sequence number.

/// Add a delete manifest entry.
    pub fn delete(&mut self, mut entry: ManifestEntry) -> Result<()> {
        entry.status = ManifestStatus::Deleted;
        entry.snapshot_id = Some(self.snapshot_id);
        self.add_entry(entry)?;
        Ok(())
    }

[liurenjie1024]

I'm not convinced. If we ask user to provide ManifestEntry, it would be confusing to user which part will be used and which part not. I think the style in java would be more clear from a user's view. If we to use ManifestEntry approach, we must have clear documentation about the behavior of each part, e.g. which is ignored, which is reserved.

[ZENOTME]

If we to use ManifestEntry approach, we must have clear documentation about the behavior of each part, e.g. which is ignored, which is reserved.

I agree with this. Then I think these functions can be pub(crate) to ensure public users will not use it. I think for now there is no demand that user need to use this API.🤔

[liurenjie1024]

I have concerns with this api, since it's error prone. According to iceberg's spec, each manifest file should contains one type of data file: data or deletes. It's quite possible that the user pass different kinds entries in previouse method, then the metadata is different. My suggestion is to follow java/python's approach:

1. A factory method like

pub fn new_v1_writer(...) {}
pub fn new_v2_writer(...) {}
pub fn new_v2_delete_writer(...) {}

2. We could use things like trait or enum to abstract out common parts of different writers.

[ZENOTME]

We could use things like trait or enum to abstract out common parts of different writers.

Difference between v1, v2, delete is:

1. the metadata of avro file
2. avro schema
3. content type
4. check in add_entry to make sure entry.content_type == writer.content_type
5. serialize the ManifestEntry

I think both differences except for serializing the ManifestEntry can be implemented by storing different data in the writer when we create the writer using the factory method. So do we really need to abstract out common parts of different writers now?🤔

[liurenjie1024]

I'm fine without trait/enum, the focus is factory methods to ensure api safety.