Btrfs or how to lose space

On the linux-btrfs mailinglist there was an interesting entry about internal fragmentation in btrfs.

A basic test brought up many questions about the btrfs filesystem design. Btrfs is a filesystem that uses the b-tree algorithm. There has been a debate if it is a good idea to use b-trees for filesystems.

I'm not enough into algorithms, but I let you decide...

The test consists of a loop, creating as much 2k sized files as possible on a 1GB Filesystem:

# for i in $(seq 1000000); \
do dd if=/dev/zero of=/mnt/file_$i bs=2048 count=1; done
(terminated after getting "No space left on device" reports).

The result from Edward Shishkin (RedHat) was 59480 Files. This would give us 2048*59480 ~ 116MB. Or in other words, we would waste around 880MB of Space.

In the meanwhile, Chris Mason, inventor of btrfs created a patch for increased utilisation.

He was able to achieve 106894 Files. That's 208 MB or a waste of 800MB. I'm not sure what he meant by the comment about the duplicate of metadata. Maybe, If you took the duplicates away, you're able to store as twice as much data on it (putting data at risk?). It still wastes almost 60% of the space...

Next on the list was of course to see how ZFS behaves.

I've created a testpool from a 1GB File. The usable space according to zfs list is 984M. I was able to squeeze in 444555 files. This results in 868MB of used space.

If we compare this to the initial 1GB capacity, we lose about 13% for metadata etc. I think that's not much, if we think of all the validation and checksumming happening behind the curtains...

Now, someone might say, nobody stores that many small files. On our mail platform we do. So any wasted space cost $$$ in the end.

Comments

Matth said…
> There has been a debate if it is a good
> idea to use b-trees for filesystems.
>
> I'm not enough into algorithms, but I let you decide...

I am into algorithms, and that sentence alone is enough to make me stop read your article. Sorry, sounded interesting otherwise.
June 22, 2010 at 9:03 AM
Unknown said…
@Matth:

I prefer to be honest about things I'm not an expert in.

But tell me, why would I need to be an expert in algorithms, if all I want to do is using a filesystem?

Fact is, btrfs is not ready for environments with many small files. And if you read the btrfs mailing-list, there is no quick fix around.
June 22, 2010 at 9:42 AM
Post a Comment

Popular posts from this blog

SLOG Latency

Image
After reading Brendan's newest blog entry, I was curious about what kind of slog latency we can see for our data migration load. To remind you, only synchronous writes go into the slog SSD devices. As this is a migration running, we can see mostly NFS write operations: In our configuration the SSD slog is mirrored (HDD4 and HDD8). Hence the same number of IOPS: The next picture shows us the latency for our SDD SLOG Device HDD 4. We can see here that latencies start at 79 us and are mostly under 200 us. There are some outliers, but approx. 95% are under 500 us: This matches quite well with the values Brendan blogged about (137-181 us), which includes NFSv3 latency. For reference (no picture here), we can see latencies of about 170-500 us mostly for NFSv4. By the way. SLOG Devices are mostly one way devices, as shown here. Only if things go really bad, they are read from...
Read more

Heating up the Data Pipeline (Part 1)

Image
Pre-Processing Data I often hear the question from our customers, how data can be transformed prior to indexing in Splunk. Damien from  Baboonbones  has done a tremendous job in creating add-ons   providing custom inputs for Splunk. Most of his custom inputs provide the means to pre-process data by allowing custom event handlers to be written. Sometimes you still want to pre-process data that gets collected from Splunk's standard input types, like file monitors, Windows EventLogs, scripted inputs etc. Also, not everyone is capable of writing custom event handlers. A requirement these customers have, is that they have rolled out a large number of Splunk Universal Forwarders and they do not want to install another agent. To summarize, the solution capable of pre-processing data, should be easy to use, be easily integrated and be build on top of their existing architecture. How to plumb Splunk Pipelines Splunk has its own fittings to connect a Universal Forwar
Read more

Heating up the Data Pipeline (Part 2)

Image
In Part 1 we went through how to route events from Splunk to a 3rd party system without losing metadata. Now I'll show you how events can be transformed using Apache NiFi and be sent back to Splunk into the  HTTP Event Collector . Note: The following is not a step-by-step documentation. To learn how to use Apache NiFi you should read the G etting Started Guide . Simple Pass-Through Flow As a first exercise we will create a simple flow, that only passes data through NiFi, without applying any complex transformations. The following picture shows a high-level NiFi flow, that receives events in our custom uncooked event format with a TCP listener, then sends the data further into a transformation "black box" (aka Processor Group), which emits events in a format, that can be ingested into a Splunk HTTP Event Collector input. Apache Nifi currently provides a rapidly growing number of processors (currently 266), which can be used for data ingestion, transfo
Read more