In this post we’ll see a Java program to write a file in HDFS. You can write a file in HDFS in two ways-
- Create an object of FSDataOutputStream and use that object to write data to file. See example.
- You can use IOUtils class provided by Hadoop framework. See example.
Writing a file in HDFS using FSDataOutputStream
package org.netjs;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class HDFSFileWrite {
public static void main(String[] args) {
Configuration conf = new Configuration();
FSDataInputStream in = null;
FSDataOutputStream out = null;
try {
FileSystem fs = FileSystem.get(conf);
// Input & Output file paths
Path inFile = new Path(args[0]);
Path outFile = new Path(args[1]);
// check if file exists
if (!fs.exists(inFile)) {
System.out.println("Input file not found");
throw new IOException("Input file not found");
}
if (fs.exists(outFile)) {
System.out.println("Output file already exists");
throw new IOException("Output file already exists");
}
in = fs.open(inFile);
out = fs.create(outFile);
byte buffer[] = new byte[256];
int bytesRead = 0;
while ((bytesRead = in.read(buffer)) > 0) {
out.write(buffer, 0, bytesRead);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}finally {
try {
if(in != null) {
in.close();
}
if(out != null) {
out.close();
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
In order to execute this program you need to add the class path to the Hadoop’s classpath.
export HADOOP_CLASSPATH=<PATH TO .class FILE>
To run program- hadoop org.netjs.HDFSFileWrite /user/process/display.txt /user/process/writeFile.txt
Writing a file in HDFS using IOUtils class
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
public class HDFSFileWrite {
public static void main(String[] args) {
Configuration conf = new Configuration();
FSDataInputStream in = null;
FSDataOutputStream out = null;
try {
FileSystem fs = FileSystem.get(conf);
// Input & Output file paths
Path inFile = new Path(args[0]);
Path outFile = new Path(args[1]);
// check if file exists
if (!fs.exists(inFile)) {
System.out.println("Input file not found");
throw new IOException("Input file not found");
}
if (fs.exists(outFile)) {
System.out.println("Output file already exists");
throw new IOException("Output file already exists");
}
in = fs.open(inFile);
out = fs.create(outFile);
IOUtils.copyBytes(in, out, 512, false);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}finally {
IOUtils.closeStream(in);
IOUtils.closeStream(out);
}
}
}
That's all for this topic Java Program to Write File in HDFS. If you have any doubt or any suggestions to make please drop a comment. Thanks!
>>>Return to Hadoop Framework Tutorial Page
Related Topics
You may also like-