Phoenix Filecube: Rise From the Ashes

Table of Contents

Amazingly, this was something I got for free… kinda. Meet a Netgear ReadyNAS RN10400, a 4-bay utter piece of crap that has all the processing power of a BeagleBone Black.

Let’s just get into it.

Acquisition

A friend who had upgraded their NAS was selling off his previous hard drives, four 4TB WD Reds, for a total of 16 TB of NAS-rated storage, not bad. Additionally he threw in this NAS, on the condition that it was broken. If I could get it to work, I could keep it. Within a few hours it was on my desk, and ready… barring one modification I had to make.

Specs

Not good. I know the RN10x series is their (now discontinued) lowest-level series, but… This unit features an unknown ARM processor, 32 bits, running at 1.2 GHz, single core, no hyperthreading, 512 MB of RAM, dual gigabit NICs, two USB 3.0 ports, one USB 2 port, and one eSATA port. The NAS will take 4 GB out of your disk array for its own OS, and 1 GB for swap space.

I will seriously compare that to a BeagleBone Black single-board computer, one of these things:

…Because I actually have one.

They have a 1 GHz 32-bit unknown core count processor, 512 MB of DDR3 RAM, and a 4 GB eMMC memory chip.

Yeah, those two things are strikingly similar in specs. To be brutally honest, I can’t do anything with that NAS to extend it. Everything maxes the CPU. Netdata, Zabbix monitoring agent, Syncthing, Disk / net I/O of 10 MB/sec…

If you’re an md person, the maximum it can resync a drive is about 17 M/sec. But we’ll get to that later.

Repair

The issue was simple, the OS was corrupt. Netgear (a very consumer-level company) has a few tools for making recovery USB devices, and I had one set up and re-install the OS, then the NAS booted into safe mode (no drives configured). The issue is that it was always attempting to boot to USB. Looking at the front, you can see a small button labeled “backup”. This button, when pushed, would run through all backup jobs configured in the web interface from top to bottom where they’re set to be bound to said button. Its secondary uses include initiating an MTP backup of a device plugged into the nearest USB port, and… booting to USB on boot. For some reason it was apparently stuck on. Nothing else really complained but at boot it was always trying USB minus the two times I got it working before I “fixed” this.

The fix was… let’s just say a hardware tweak. The inside of the NAS is actually really well constructed, a four port SATA backplane connected to the mainboard through a PCIe 1x slot, the mainboard connecting to the front-panel power button and LEDs through some pin headers. Both the power and backup buttons there have two physical momentary switches behind them, and I decided to remove the backup’s since.. well all three features I can access through other means anyways.

After taking it all apart (really easy since all screws are identical in similar sections, all case screws are the same, all backplane, all mainboard… And the actual case itself is a few metal pieces that slide together), I checked with a meter.. one of the two buttons there was making near-constant contact. Time to operate.

Attempt 1: Desoldering

Requirements:

• A clean disassembly
• Soldering iron
• Small enough micro-soldering tip
• Very fine rosin-core solder
• Desoldering braid
• Magnification
• Cat-proofing the table from curious felines
• Patience

Result: no dice. I do not know what that PCB is made of but it wasn’t taking any heat. I could melt any solder I put down just fine, but anything from the factory wouldn’t budge and was even still cool to the touch right after melting new solder right off it. After 45 minutes I really said out lout (to nobody) “Please tell me I didn’t spend all this time when some pliers would have just worked”

Attempt 2: Pliers

Requirements:

• Strength
• Needle-nose pliers
• Urge to destroy

Result: Two pieces of small bent momentary switches to dispose of, and one slightly more naked circuit board. Somehow, I couldn’t even desolder the broken off leads, which I had assumed would have given me a better chance. Either way, I just had to hope when I put everything together that the unit still worked and that the problem was solved.

Aftermath

Well, it’s running in those pictures, meaning that I have 1/1 successes, meaning my fix has a 100% success rate, I’m happy with those numbers (😆).

Anyways let’s get into some more tech stuff.

Features

The NAS itself is built off of Debian of some sort, the SSH version string says Debian-5+deb8u7.netgear1 while the apt repos are debian.org Jessie.

The web interface is slow (to be expected) but functional, and easy to use.

Storage

The unit itself has 4 bays, which can be configured into one logical volume of either active RAID members, hot spares, or, if you have adapters to fit SSDs into 3.5" bays, tiered storage, with “X-RAID” active. The alternative, Flex-RAID, can support more than one volume, however, vertical expansion (that is, swapping drives for larger capacity ones) can’t be used without re-creating the array (and losing its contents).

The bays themselves use hot-swappable drive cages with plastic inserts instead of screw mounts, really cool since you have no screws to keep track of, but the latch that releases the handle also releases the lock for the insert, meaning you have to be careful that you aren’t pulling out the cage while the drive stays in place.

The device uses md (Linux kernel-level RAID) for RAID supporting all the common levels, and on top of this creates a BTRFS filesystem. The volume is divided into “shares” with dedicated owners, allowable access protocols, snapshot schedules, and quotas. Shares can also be configured to allow Copy-on-Write (CoW) and on-disk transparent compression, both prominent BTRFS features.

Additionally, (file-level) iSCSI LUNs can be created and served out… more on that later.

Snapshots, being done by BTRFS, only take up disk space relative to the amount of data that’s changed since the snapshot has taken. This means that, when taken, they have a size of 0, and appear to increase as the snapshot holds on to “old” data that can’t be freed back up once unused. Two snapshots of 10 GB of data here are only taking about 4 GB total since that’s all that’s changed.

SSDs inserted become “tiered storage” where highly-accessed files will be moved to the SSD instead of the disk array. For comparison with ZFS, it’s L2ARC, essentially. Just a giant read cache.

Network and Protocols

The device has two gigabit NICs that can be bonded using a few modes like LACP or a few others that don’t require dedicated switches.

As for protocols, It can deliver SMB3 (with transport encryption) with MacOS enhancements, AFP, NFS (including NFSv4), FTP (With FTPS support and rate limiting), act as an iTunes server, ReadyDLNA, keep an active Rsync daemon, HTTP / HTTPS (WebDAV), keep an SSH server running, and finally, export some (very minimal, really just RFC1213-MIB) SNMP statistics (v1 or v2c only).

Additionally it can support backing up files from USB devices over MTP, and act as a Time Machine server for MacOS (which is stupidly simple to set up yourself on a spare machine, and something I might show later in another post).

Apps

Like most similar devices, it has Apps that can add extra features, like a database server, more monitoring tools, what have you. Given it’s limited resources, really none of them are actually viable.

Backup Capabilities

Create an unlimited number of backup jobs, each with their own schedules, that can be triggered individually or with the “Backup All” button in the interface that runs the list top to bottom.

Backups can run from a local share, one of the three USB ports, or the eSATA port, or from a remote device using FTP, NFS, Rsync, Rsync over SSH, or SMB via the timestamp or archive bit, and can transfer to the same set of endpoints.

Yes, this means that with a few backup jobs you can use the front-panel button (if you didn’t remove it like I did) or the web interface button to backup network devices to the NAS and then send the NAS data to an off-site connection. Note that backups of shares are only one share at a time, you need multiple jobs to backup multiple shares.

Or with a fast enough device (or small enough dataset), send everything in a share to an external eSATA or USB drive.

Misc

Allows disk spin-down when they’re not in use, either always or on scheduled times, allows setting a timer for power-on / power-off throughout the week, Wake-on-LAN, and UPS monitoring via SNMP (v1 or v2c only, no manual selection).

My Opinions

Really, for low-end consumer, it’s a good NAS. For the level that I operate at, it’s severely lacking but something that, now that I have it, is something that I’m not giving up anytime soon (at least to upgrade). 12 TB of space in it (for now) with one-disk redundancy is way more than I had before (1 TB with a FreeNAS VM), even though one of the disks I bought is currently at… 808 reallocated sectors, 25 ATA errors, 3.8 years of power-on hours, and… it’s still working despite me scrambling to order a new one ASAP, seems it lived a good life, just old. Unfortunately due to the processor, it takes 3 days to resync the volume when md isn’t sure about something, and would take closer to half a month at minimum to perform a BTRFS defrag, rebalance, or scrub. Along with that, it’s obviously not really made for this large of a dataset, 12 TB is just more than it can reasonably handle… but given the age and target market, I can kinda understand that.

Of which, BTRFS. Love it or hate it, I’ve used it for quite a while and had no issues. The stability issues are a thing of the past, though I wouldn’t use it’s filesystem-level raid5 just yet. This device really doesn’t have the power to use ZFS (like FreeNAS does), but it also maintains a lot of the good features: copy-on-write, instant overhead-free snapshots, on-disk compression, and a good deal of filesystem-level error checking.

By the way, fun fact: The 4 GB system partition is a RAID-1 of three of the disks, and the 1 GB swap partition is a RAID-10 of all four. Yes, my swap needs RAID-10, apparently. Ask Netgear.

And as for resources, I really find it easier to run one (or two) higher-powered VMs that just have some NFS mounts in /etc/fstab, maybe configure some mount and target systemd units to shut down the dependent services if anything goes wrong. I’ll get to it in another post, but NFS is, on this network, more performant than SMB, at the cost of weird permissions stuff.

Also, the power connector is this weird, proprietary, locking thing. You need to pull back the spring-loaded plastic housing to plug or unplug, and even then it takes some force. The manual states that the official manufacturer directed way to reset the NAS from a crash, or to give it a hard reset for any reason, is to just pull the plug for a few seconds. Power button not responding? Unplug. USB boot failed? Unplug.

Really overall, for someone who’s not into as high-level (net) gear as I am and who only needs a terabyte or two at most for their storage, it would be fine. Cheap, and it shows, but fine. Really though… When I upgrade, and that is a when not an if, this will likely be the backup system that I push data to, in order to have a secondary copy before sending it off-site. Something with very little load, and what load it does have is just a burst-transfer.