Introduction
I have been using Signal daily to communicate with my friends and relatives for several years. It feels way more comfortable to have online discussions without my behavior being tracked and analyzed by someone else. It's a good alternative compared to big commercial companies' products and their aggressive marketing algorithms. Since these conversations are important, I enabled built-in backups during the first few months of usage. Signal exports all media files and messages as a single file encrypted with a random passphrase.
For the last few months, I have noticed strange behavior from a Signal app. It started consuming too much RAM and lagging. I am almost sure the reason is that my chat history became too big. The backup file currently is 2GB and includes 2 years of chatting. I assume Signal doesn't optimize the way how old messages are loaded. And this explains why the app has started using the same amount of operating memory as Android OS itself. Except for periodic unresponsiveness, the device was discharging rapidly during the day and rebooting from time to time. Later I plan to open an issue on GitHub to help the Signal developers solve the issue. Yesterday I decided to export my latest backup, check its integrity, and reset my account.
Below you can see a screenshot where the app consumes half the amount allocated for the operating system. Once I even saw it eating 1GB Carl!
Enable Backup
First of all, you need to have backup enabled in your settings. It's not configured by default.
Go to Settings -> Chats and media -> Chat backups
This action will display a random passphrase you have to save securely somewhere. I personally use KeepassX. You can not reset this passphrase.
Done. Now Signal will back up data daily.
Decrypt Signal Data
The Signal team does not provide an official tool to decrypt a backup file on your PC. By design, backups are made to move them to another phone or recover them after reinstalling the app. But luckily I found online a command-line tool for this purpose. You can find this open-source project following the GitHub link.
Go to the Releases page and download a binary file for your operating systems. For my machine, it's signal-back_linux_amd64. And run:
mv ~/Downloads/signal-back_linux_amd64 ~/signal-back
chmod +x ~/signal-backCopy a backup file from your phone to your PC. On Android phones, it's located under the Signal/Backups folder under the root directory.
1. To export all messages to a CVS file run:
./signal-back format -f CSV -o backup.csv signal-XXX.backup2. To export all media files run:
./signal-back extract -o signal-media signal-XXX.backup3. To export all SMS as an XML file to restore later with SMS Backup & Restore apps run:
./signal-back format -f XML -o backup.xml signal-XXX.backupThanks to username @xeals on GitHub decryption is super easy. But even at this point, I encountered a small problem I want to tell you about below.
Resource problem during Decryption
As it turns out signal-back tool doesn't make any RAM optimization. Currently, I have a laptop with only 8GB RAM and I failed to decrypt a file on my machine. The tool was throwing out memory errors. I think the program reads the file, decrypts it all, and then writes to a CSV file, without checking if OS can allocate the required RAM amount. My machine hung up a few times. On the next execution, I ran top to make sure it was a problem with RAM.
I tried to limit the RAM amount allocated to the signal-back process with ulimit. ulimit is a Linux utility to restrict resource amounts for processes. Below is the command I used:
ulimit -Sv 6000000My hope was that the software respects OS limits and adapts to a limited RAM amount. But after setting the limit of 6GB RAM I received the following errors:
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x5d2957, 0x16)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/panic.go:616 +0x81
runtime.sysMap(0xc509890000, 0x35a0000, 0x423f00, 0x6f4d98)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/mem_linux.go:216 +0x20a
runtime.(*mheap).sysAlloc(0x6dc760, 0x35a0000, 0x56b26c)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/malloc.go:470 +0xd4
runtime.(*mheap).grow(0x6dc760, 0x1ace, 0x0)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/mheap.go:907 +0x60
runtime.(*mheap).allocSpanLocked(0x6dc760, 0x1ace, 0x6f4da8, 0x0)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/mheap.go:820 +0x301
runtime.(*mheap).alloc_m(0x6dc760, 0x1ace, 0x7fff4f1d0101, 0x42290a)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/mheap.go:686 +0x118
runtime.(*mheap).alloc.func1()
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/mheap.go:753 +0x4d
runtime.(*mheap).alloc(0x6dc760, 0x1ace, 0x7f7d6d010101, 0x7f7d6daaf090)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/mheap.go:752 +0x8a
runtime.largeAlloc(0x359b2e1, 0x590101, 0x599c60)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/malloc.go:826 +0x94
runtime.mallocgc.func1()
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/malloc.go:721 +0x46
runtime.systemstack(0x0)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/proc.go:1175
goroutine 1 [running]:
runtime.systemstack_switch()
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/asm_amd64.s:363 fp=0xc42004d478 sp=0xc42004d470 pc=0x44f6b0
runtime.mallocgc(0x359b2e1, 0x58c820, 0x40f701, 0xc42004d550)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/malloc.go:720 +0x8a2 fp=0xc42004d518 sp=0xc42004d478 pc=0x40fd02
runtime.makeslice(0x58c820, 0x359b2e1, 0x359b2e1, 0x8, 0x68, 0xc502274230)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/slice.go:61 +0x77 fp=0xc42004d548 sp=0xc42004d518 pc=0x43c307
bytes.makeSlice(0x359b2e1, 0x0, 0x0, 0x0)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/bytes/buffer.go:230 +0x6d fp=0xc42004d588 sp=0xc42004d548 pc=0x4bd4ed
bytes.(*Buffer).grow(0xc502274230, 0x359b2e1, 0xc42004d608)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/bytes/buffer.go:144 +0x151 fp=0xc42004d5d8 sp=0xc42004d588 pc=0x4bcea1
bytes.(*Buffer).Grow(0xc502274230, 0x359b2e1)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/bytes/buffer.go:163 +0x3a fp=0xc42004d600 sp=0xc42004d5d8 pc=0x4bd04a
io/ioutil.readAll(0x5f2740, 0xc42008f730, 0x359b2e1, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/io/ioutil/ioutil.go:34 +0x93 fp=0xc42004d648 sp=0xc42004d600 pc=0x4c7983
io/ioutil.ReadFile(0xc501b8aa80, 0xd, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/io/ioutil/ioutil.go:73 +0xd5 fp=0xc42004d6a0 sp=0xc42004d648 pc=0x4c7b05
github.com/xeals/signal-back/cmd.ExtractAttachments(0xc4200f40e0, 0x20, 0x0)
/home/travis/gopath/src/github.com/xeals/signal-back/cmd/extract.go:88 +0x3d3 fp=0xc42004d9e0 sp=0xc42004d6a0 pc=0x566963
github.com/xeals/signal-back/cmd.glob..func3(0xc4200a0420, 0x0, 0xc4200a0420)
/home/travis/gopath/src/github.com/xeals/signal-back/cmd/extract.go:43 +0x8c fp=0xc42004da38 sp=0xc42004d9e0 pc=0x56b26c
github.com/xeals/signal-back/vendor/github.com/urfave/cli.HandleAction(0x593a80, 0x5daee0, 0xc4200a0420, 0xc42009a100, 0x0)
/home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:490 +0xc8 fp=0xc42004da60 sp=0xc42004da38 pc=0x4f44a8
github.com/xeals/signal-back/vendor/github.com/urfave cli.Command.Run(0x5cf260, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5d682b, 0x24, 0x5d6cae, ...)
/home/travis/gopath/src/github.com/xeals/signal-back/vendo /github.com/urfave/cli/command.go:210 +0xa36 fp=0xc42004dcd0 sp=0xc42004da60 pc=0x4f5716
github.com/xeals/signal-back/vendor/github.com/urfave/cli.(*App).Run(0xc42009c1a0, 0xc4200900f0, 0x5, 0x5, 0x0, 0x0)
/home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:255 +0x6a0 fp=0xc42004dea8 sp=0xc42004dcd0 pc=0x4f2830
main.main()
/home/travis/gopath/src/github.com/xeals/signal-back/main.go:52 +0x317 fp=0xc42004df88 sp=0xc42004dea8 pc=0x56c637
runtime.main()
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/proc.go:198 +0x212 fp=0xc42004dfe0 sp=0xc42004df88 pc=0x42a792
runtime.goexit()
/home/travis/.gimme/versions/go1.10.3.linux.amd64/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42004dfe8 sp=0xc42004dfe0 pc=0x4520a1At this point, I decided to rent a 16GB VPS to just decrypt files there. I ran top alongside the signal-back on a server. The decryption process used 9.1GB of RAM for a 2GB backup file. So if you have such large files then get a machine with FILE_SIZE * 5 amount of RAM. The backup file turned out to be healthy and I was able to reset my account. Initially, I thought this process will take a few minutes. But in the end, I spent several hours solving this problem on Saturday evening.
Conclusion
This incident made me think about calculation power in general. What if I could not rent a VPS? Then I could not decrypt the file. Or I had to optimize the tool, which is kinda impossible for me at the moment. signal-back is written in Go and I don't know this language yet. Another approach would be to open an issue on signal-back GitHub. But even then the developer may not implement the fix, may take a lot of time, or technically be impossible because of the backup design in the Signal app. The cloud was not possible a few years ago and I felt all the pain scientists had in past. When you want to do a calculation on some data, answer your complex questions but you don't have enough resources. It's crazy. And to be honest I am happy today that this isn't a problem anymore. This small technical journey reminded me how fascinating and complex the information world is.