How to split log file with offset value in unix?
I have a really large log file (9 GB - I know I need to fix this) on my box. I need to split into chunks so that I can upload it to Amazon S3 for backup. The maximum S3 file size is 5 GB. So I would like to split this into several chunks and then download each one.
Here's the catch, I only have 5GB on my server for free, so I can't just do a simple unix split. Here's what I want to do:
- grab the first 4GB of the log file and spit it out into a separate file (call it segment 1)
- Load this segment1 into s3.
- rm segment1 to make space.
- take the average 4GB from the log file and upload it to s3. Cleaning still
- Take the remaining 1GB and upload it to S3.
I can't seem to find the right unix command to split with offset. Split only does things in equal chunks, and csplit doesn't seem to have what I need. Any recommendations?
a source to share
One (collapsed) solution is to compress it first. The log text file should easily go from 9G to below 5G, after which you delete the original, giving you 9G of free space.
Then you transfer this compressed file directly through split
to avoid using up more disk space. You will end up with a compressed file and three files to download.
Download them, then delete them, then unzip the original log.
=====
The best solution is to just count the lines (say 3 million) and use an awk script to fetch and send the individual pieces:
awk '1,1000000 {print}' biglogfile > bit1
# send and delete bit1
awk '1000001,2000000 {print}' biglogfile > bit2
# send and delete bit2
awk '2000001,3000000 {print}' biglogfile > bit3
# send and delete bit3
Then, on the other hand, you can process bit1
through bit3
individually or recombine them:
mv bit1 whole
cat bit2 >>whole ; rm bit2
cat bit3 >>whole ; rm bit3
And, of course, this separation can be accomplished using any of the standard text processing tools in the Unix: perl
, python
, awk
, head/tail
combo. It depends on what you're comfortable with.
a source to share
First gzip -9 your log file.
Then write a small shell script to use dd:
#!/bin/env sh
chunk_size = 2048 * 1048576; #gigs in megabytes
input_file = shift;
len = `stat '%s' $input_file`
chunks = $(($len/$chunk_size + 1))
for i in {0...$chunks}
do
dd if=$input_file skip=$i of=$input_file.part count=1 bs=$chunk_size
scp $input_file.part servername:path/$input_file.part.$i
done
I just flopped this off my head, so I don't know if this would work unchanged, but something very similar to what you need.
a source to share
You can use dd. You will need to specify bs (memory buffer size), skip (number of buffers to skip) and count (number of buffers to copy) in each block.
Thus, using a 10Meg buffer size you should:
# For the first 4Gig
dd if=myfile.log bs=10M skip=0 count=400 of=part1.logbit
<upload part1.logbit and remove it>
# For the second 4Gig
dd if=myfile.log bs=10M skip=400 count=400 of=part2.logbit
...
You can also use compression of the data you intend to transfer:
dd if=myfile.log bs=10M skip=800 count=400 | gzip -c > part3.logbit.gz
There may be friendlier methods out there.
dd has some real disadvantages. If you are using a small buffer size, it is much slower. But you can only skip / search in the file with multiple bs. So if you want to start reading data from a simple offset, you are in a real fiddle. I am distracted anyway.
a source to share