Weekend Project: Get to Know Btrfs

The Butter/Better/B-tree Filesystem, Btrfs, is supposedly destined to become the default Linux filesystem. What makes it special, and what's wrong with good old tried-and-true Ext2/3/4?

Linux supports a gigantic number of filesystems: removable media, network, cluster, cloud, journaling, virtual machine, compressed, embedded, hardware inter-connect, pseudo-filesystems that live only in memory, Mac and Windows filesystems, and many more.

You are doubtless familiar with the general-purpose Ext2/3/4, JFS, XFS, and Reiser filesystems that we use on our desktop PCs and servers. With all of these filesystems cluttering up the landscape, what is the point of yet another one? (There is even YAFFS: Yet Another Flash File System.)

The point is meeting new needs and workloads, and building functionality into the filesystem rather than relying on a herd of external utilities. Btrfs is rather like a blend of features from ReiserFS and ZFS, Sun's advanced copy-on-write/volume manager/RAID/snapshot/etc. filesystem.

Many Linux users yearn for a native port of ZFS, but its GPL-incompatible license (the Sun CDDL) ensures that Sun's implementation (now Oracle's) can't be included in the Linux kernel.

Even so, you can't keep a good hacker down, and so there are two ports for Linux. One is ZFS on FUSE, which runs ZFS in user-space. It's included in a lot of distros so it's an easy installation. The other one is ZFS on Linux. This is a build of ZFS as a kernel module for users to install, and so you get kernel support without a GPL violation because it is not distributed with the kernel.

It's great having those options to try out ZFS, and I applaud the maintainers of these ZFS projects. Still, it looks like Btrfs is going to take the place that ZFS could have owned were it not for its incompatible license. Oracle is the primary sponsor of Btrfs, and plans to make it the default filesystem in Oracle Unbreakable Linux sometime in 2012. Btrfs isn't just an Oracle project, but has a lot of community support from the Linux kernel team and many Linux distributions. Odds are it's included in your favorite distro. (Run cat /proc/filesystems to see what filesystems your Linux supports.)

So what does this amazing super-duper filesystem do? How about a handy bullet-pointed list to answer this question?

RAID 0, 1, 10COWIncremental backupOnline defraggzip and LZO compressionSpace-efficient packing of small filesDynamic inode allocationChecksums on data and metadataShrink and grow storage volumesExtentsSnapshots16 EiB maximum file size

Planned features include RAID 5 and 6, deduplication, and a ready-for-primetime filesystem checker, btrfsck. You can try out btrfsck now because it is included in btrfsprogs. (Which of course Debian/Ubuntu/Mint etc. changes to btrfs-tools, and Fedora calls it btrfs-progs.) But it is not ready for production systems yet.

Putting the finishing touches on btrfsck is the last big step before Oracle makes it the default filesystem in their next Unbreakable Linux release. Fedora 16 Linux was supposed to default to Btrfs, but now they're aiming for Fedora 17 in May 2012.

I'm a big fan of RAID 10, which is RAID 1+0, mirroring and striping. It is expensive of disks because only 50% of your total disk capacity goes to storage. But it is simple, robust, and fast. Half your disks can fail without losing your data. I got burned out on RAID 5 and 6 years ago; perhaps I had bad RAID mojo, but I experienced a lot of failures, and they are slow. It seemed the systems under my care were more adept at propagating parity errors than operating correctly. So for me, RAID 5 and 6 can sit on the back burner indefinitely as long as I have RAID 10.

16 EiB is exbibytes, a measurement close to the more commonly used exabyte. An exbibyte is 1,024 pebibytes. In comparison Ext4 maxes out at volumes with a maximum size of one exbibyte and file sizes up 16 tebibytes. However you say it, it is a lot.

Btrfs doesn't contain any database-specific optimizations, and is not a clustering filesystem. It is designed to handle very large storage volumes, protect data, simplify large storage management, and read and write fast.

A COW – copy on write – filesystem is extra-careful with writing your data. When you make a change to a file, the old data are not overwritten. Instead, the filesystem allocates new blocks for the new data, and only the changed data are given a new allocation. The downside is this creates fragmentation. So Btrfs supports online defragmentation with the

btrfs filesystem defragment

command.

COW filesystems lend themselves to easy, efficient snapshots, and Btrfs supports both snapshots and rollbacks. The easy safe way to try Btrfs is to create a new partition for testing. Gparted supports Btrfs, as you can see in figure 1.

Next, mount this partition. In this example the mountpoint is /btrfs-volume:

# mount -t btrfs /dev/sda8 /btrfs-volume

Now we can create a subvolume in this partition. Subvolumes are cool. They are like independent filesystems inside the parent filesystem, with their own mountpoints and options. Create one this way:

# btrfs subvolume create btrfs-volume/test

And that's all there is to it. You'll see this as an ordinary directory in your file manager (figure 2). You don't need to worry about allocating space like you do with normal disk partitions, because subvolumes automatically snag whatever space they need from the parent volume as you add data to them. So you can go ahead and copy some files into the test subvolume. You'll need root permissions, or you can futz with the file permissions in the usual way and change them to an unprivileged user.

Now let's create a snapshot:

# btrfs subvolume snapshot btrfs-volume/test btrfs-volume/test-snapshot-1Create a snapshot of 'btrfs-volume/test' in 'btrfs-volume/test-snapshot-1'

Snapshots are very efficient because multiple snapshots share the same original files and copy only the changes. You can list all the snapshots in the same volume; you need to name one of them and then all of them are displayed:

# btrfs subvolume list btrfs-volume/testID 256 top level 5 path testID 257 top level 5 path test-snapshot-1

This also shows that Btrfs sees snapshots and subvolumes as the same things. Your snapshots can be copied elsewhere as backups, or mounted independently to different mountpoints. Want to roll back to an earlier snapshot? First set the snapshot as the default. You need the snapshot ID, and then the path:

# btrfs subvolume set-default 257 btrfs-volume/

Then unmount the subvolume, and then remount:

# umount btrfs-volume# mount -t btrfs /dev/sda8 btrfs-volume

Is that not cool? After creating subvolumes you don't need to mount the parent volume.

Btrfs is still rough around the edges, and the documentation and administration tools are incomplete. If you've used ZFS then Btrfs feels like a clunky copy, because administering ZFS is faster and easier. ZFS has a several-year head start on Btrfs, though. I expect Btrfs will improve rapidly as it becomes more widely used.

Learn more about Btrfs with The Linux Foundation's free Linux training tutorial Introduction to Btrfs. A full course schedule of Linux SysAmdin training is also available.

Comments (4)
You must be logged in to post a comment. Please register if you do not have an account yet.

View the original article here

Organizing Open Source Efforts at NASA

"When I think of open source, Linux is the core," says William Eshagh, a technologist working on Open Government and the Nebula Cloud Computing Platform out of the NASA Ames Research Center. Eshagh recently announced the launch of code.nasa.gov, a new NASA website intended to help the organization unify and expand its open source activities. Recently I spoke with Eshagh and his colleague, Sean Herron, a technology strategist at NASA, about the new site and the roles Linux and open source play at the organization.

Eshagh says that the idea behind the NASA code site is to highlight the Linux and open source projects at NASA. "We believe that the future is open," he says. Although NASA uses a broad array of technology, Linux is the default system and has found its way into both space and operational systems. In fact, the websites are built on Linux, the launch countdown clock runs on Fedora servers, and Nebula, the open-source cloud computing project, is Ubuntu based. Further, NASA worked with Rackspace Hosting to launch the OpenStack project, the open source cloud computing platform for public and private clouds.

Why is NASA contributing to open source? Eshagh says that NASA's open systems help inspire the public and provide an opportunity for citizens to work with the organization and help move its missions forward. But the code site isn't only about sharing with the public and making NASA more open. The site itself is intended to help NASA figure out how the organization is participating in open source projects.

In the initial phase, the code site organizers are focusing on providing a central location to organize the open source activities at NASA and lower the barriers to building open technology with the help of the public. Herron says that the biggest barrier is that people simply don't know what's going on in NASA because there is no central list of open source projects or contributions. Within NASA, employees don't even have a way to figure out what their colleagues are working on, or who to talk to within the organization about details such as open source licenses.

At NASA, the open source process starts with the Software Release Authority, which approves the release of software. Eshagh says that even finding the names of the people in the Software Release Authority was an exercise, so moving the list of names out front and shining a light on it makes it easier to find the person responsible. The new guide on the code site explains the software release guidelines and provides a list of contacts and details about releasing the software, such as formal software engineering requirements.

Phase two of the code project is community focused and has already started. Eshagh says there's a lot of interest in open source at NASA, including internal interest, but the open.NASA team is still trying to figure out the best way to connect people with projects within the agency.

Eshagh says that the third phase, which focuses on version control, issue tracking, documentation, planning, and management, is more complicated. He points to the Goddard General Mission Analysis Tool (GMAT), an open source, platform independent trajectory optimization and design system, as an example. There is no coherent or coordinated approach to develop software and accept contributions from the public and industry. "What services do they need to be successful? What guidance do they need from NASA?," Eshagh wonders. "We're trying to find best-of-breed software solutions online; GitHub comes to mind," he says.

Phase three also will include the roll out of documentation systems and a wiki, which Eshagh and his team want to offer as a service, but in a focused, organized way to help projects move forward. He says that NASA doesn't promote any project or product – they just want the best tool for the job. "We're taking an iterative approach and making information available as we get it and publish it," Herron says. They've already received a bunch of feedback about licenses, for example.

How will the open.NASA team measure the success of the code site and their other efforts? "We're trying to build a community," Eshagh explains. "We've kind of tapped into an unsatisfied need for public and private individuals to come together."

He says they'll measure success by how many projects that they didn't previously know about come forward and highlight what they are doing. "People are actually reaching out and I think that's a measure of success," he says. Also, the quantity and quality of the projects and toolchains, as well as how many people use them, will be considerations.

In December, Eshagh announced NASA's presence on GitHub, and their first public repository houses NASA's World Wind Java project, an open source 3D interactive world viewer. Additional projects are being added, including OpenMDAO, an open-source Multidisciplinary Design Analysis and Optimization (MDAO) framework; NASA Ames StereoPipeline, a suite of automated geodesy and stereogrammetry tools; and NASA Vision Workbench, a general-purpose image processing and computer vision library.

In March 2011, NASA hosted its first Open Source Summit at Ames Research Center in Mountain View California. GitHub CEO Chris Wanstrath and Pascal Finette, Director of Mozilla Labs, were among the speakers. Eshagh says that at the event, he learned that when Erlang was first released on GitHub, contributions increased by 500 percent. "We are hoping to tap into that energy," he adds.

"A lot of our projects were launched under SVN and continue to be operated under there," Eshagh says. Now open.NASA is looking at git-svn to bridge these source control systems.

"A lot of projects don't have change history or version control, so GitHub will help with source control and make it visible and available," Herron adds. Since they've posted the GitHub projects, Eshagh and his team have already seen some forks and contributions back, but he says the trick is to figure out how to get the project owners to engage and monitor the projects or to move to one system.

Which NASA projects would Eshagh like to see added to GitHub? "We have so many projects, we don't have favorites," he says. "If this is a viable solution, increases participation, and makes it easier for developers to develop, then we'd like to see them there." He adds that his team would like to see all of NASA's open source projects have version control, use best practices, and be handled in a way that the public can see them.

At the end of 2011, Nick Skytland, Program Manager of Open Government at the Johnson Space Center, posted the 2011 Annual Report by the NASA Open Government Initiative. His infographic says that there were 140,000,000 views of the NASA homepage; 17 Tweetups held with more than 1,600 participants; 2,371,250 combined followers on Twitter, Facebook, and Google+; and 50,000 followers on Google+ within the first 25 days.

"The scope and reach of our social media is not insignificant," Eshagh says. Herron points out that the nasa.gov site is the most visited US government site. He says that the community is very engaged. "People love to see our code," he adds. "People are excited about it." In fact, the open source team hopes to use their code to keep people excited about the space program.

NASA is now in an interesting phase, Eshagh explains. He says that after the space shuttle program ended last year, NASA Deputy Administrator Lori Garver was speaking to a group of students when one of them asked her whether she's now out of a job. (She's not.) The new code site helps illustrate how many projects are still active and growing at NASA. "We still have a lot of work to do and a lot of people are pulling for us," Eshagh says.

Comments (0)
You must be logged in to post a comment. Please register if you do not have an account yet.

View the original article here

Weekend Project: Learning Ins and Outs of Arduino

Arduino is an open embedded hardware and software platform designed for rapid creativity. It's both a great introduction to embedded programming and a fast track to building all kinds of cool devices like animatronics, robots, fabulous blinky things, animated clothing, games, your own little fabs... you can build what you imagine. Follow along as we learn both embedded programming and basic electronics.

Arduino was invented by Massimo Banzi, a self-taught electronics guru who has been fascinated by electronics since childhood. Mr. Banzi had what I think of as a dream childhood: endless hours spent dissecting, studying, re-assembling things in creative ways, and testing to destruction. Mr. Banzi designed Arduino to be friendly and flexible to creative people who want to build things, rather than a rigid, overly-technical platform requiring engineering expertise.

The microprocessor revolution has removed a lot of barriers for newcomers, and considerably speeded up the pace of iteration. In the olden days building electronic devices means connecting wires and components, and even small changes were time-consuming hardware changes. Now a lot of electronics functions have moved to software, and changes are done in code.

Arduino is a genuinely interactive platform (not fake interactive like clicking dumb stuff on Web pages) that accepts different types of inputs, and supports all kinds of outputs: motion detector, touchpad, keyboard, audio signals, light, motors... if you can figure out how to connect it you can make it go. It's the ultimate low-cost "what-if" platform: What if I connect these things? What if I boost the power this high? What if I give it these instructions? Mr. Banzi calls it "the art of chance." Figure 1 shows an Arduino Uno; the Arduino boards contain a microprocessor and analog and digital inputs and outputs. There are several different Arduino boards.

You'll find a lot of great documentation online at Arduino and Adafruit Industries, and Mr. Banzi's book Getting Started With Arduino is a must-have.

The world is over-full of useful garbage: circuit boards, speakers, motors, wiring, enclosures, video screens, you name it, our throwaway society is a do-it-yourselfer's paradise. With some basic skills and knowledge you can recycle and reuse all kinds of electronics components. Tons of devices get chucked into landfills because a five-cent part like a resistor or capacitor failed. As far as I'm concerned this is found money, and a great big wonderful playground. At the least having a box full of old stuff gives you a bunch of nothing-to-lose components for practice and experimentation.

The Arduino integrated development environment (IDE) is a beautiful creation. The Arduino programming language is based on the Processing language, which was designed for creative projects. It looks a lot like C and C++. The IDE compiles and uploads your code to your Arduino board; it is fast and you can make and test a lot of changes in a short time. An Arduino program is called a sketch. See Installing Arduino on Linux for installation instructions.

You will need to know how to solder. It's really not hard to learn how to do it the right way, and the Web is full of good video howtos. It just takes a little practice and decent tools. Get yourself a good variable-heat soldering iron and 60/40 rosin core lead solder, or 63/37. Don't use silver solder unless you know what you're doing, and lead-free solder is junk and won't work right. I use a Weller WLC100 40-Watt soldering station, and I love it. You're dealing with small, delicate components, not brazing plumbing joints, so having the right heat and a little finesse make all the difference.

Another good tool is a lighted magnifier. Don't be all proud and think your eyesight is too awesome for a little help; it's better to see what you're doing.

Adafruit industries sells all kinds of Arduino gear, and has a lot of great tutorials. I recommend starting with these hardware bundles because they come with enough parts for several projects:

Adafruit ARDX – v1.3 Experimentation Kit for Arduino This has an Arduino board, solderless breadboard, wires, resistors, blinky LEDs, USB cable, a little motor, experimenter's guide, and a bunch more goodies. $85.00.9-volt power supply. Seven bucks. You could use batteries, but batteries lose strength as they age, so you don't get a steady voltage.Tool kit that includes an adjustable-temperature soldering iron, digital multimeter, cutters and strippers, solder, vise, and a power supply. $100.

Other good accessories are an anti-static mat and a wrist grounding strap. These little electronics are pretty robust and don't seem bothered by static electricity, but it's cheap insurance in a high-static environment. Check out the Shields page for more neat stuff like the Wave audio shield for adding sound effects to an Arduino project, a touchscreen, a chip programmer, and LED matrix boards.

Let's talk about volts (V), current (I), and resistance (r) because there is much confusion about these. Volts are measured in voltage, current is measured in amps, and resistance is measured in ohms. Electricity is often compared to water because they behave similarly: voltage is like water pressure, current is like flow rate, and resistance is akin to pipe diameter. If you increase the voltage you also increase current. A bigger pipe allows more current. If you decrease the pipe size then you increase resistance.

Talk is cheap, so take a look at Figure 3. This is an old circuit board from a washing machine. See the stripey things? Those are resistors. All circuit boards have gobs of resistors, because these control how much current flows over each circuit. The power supply always pushes out more power than the individual circuits can handle, because it has to supply multiple circuits. So there are resistors on each circuit to throttle down the current to where it can safely handle it.

Again, there is a good water analogy — out here in my little piece of the world we use irrigation ditches. The output from the ditch is too much for a single row of plants, because its purpose is to supply multiple rows of plants with water. So we have systems of dams and diverters to restrict and guide the flow.

In your electronic adventures you're going to be calculating resistor sizes for your circuits, using the formula R (resistance) = V (voltage) / I (current). This is known as Ohm's Law, named for physicist Georg Ohm who figured out all kinds of neat things and described them in math for us to use. There are nice online calculators, so don't worry about getting it right all by yourself.

That's all for now. In the next tutorial, we'll learn about loading and editing sketches, and making your Arduino board do stuff.

Comments (0)
You must be logged in to post a comment. Please register if you do not have an account yet.

View the original article here