How to write data to Hadoop using .NET

Here is an example of .NET code that writes data to a Hadoop table using the Microsoft.Hadoop.Avro library:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Hadoop.Avro.Container;
using Microsoft.Hadoop.WebHDFS.Adapters;

namespace HadoopWriter
{
    class Program
    {
        static void Main(string[] args)
        {
            // Define the Hadoop cluster credentials and endpoint
            string clusterUri = "http://your-hadoop-cluster-uri/";
            string userName = "your-hadoop-username";
            string password = "your-hadoop-password";

            // Define the Hadoop table location
            string tableLocation = "/your-table-location/";

            // Define the Avro schema for the table
            string schema = @"{
                ""type"": ""record"",
                ""name"": ""Person"",
                ""fields"": [
                    {""name"": ""Name"", ""type"": ""string""},
                    {""name"": ""Age"", ""type"": ""int""}
                ]
            }";

            // Create a list of Person objects to write to the table
            List<Person> people = new List<Person>()
            {
                new Person { Name = "John", Age = 30 },
                new Person { Name = "Mary", Age = 25 }
            };

            // Write the list of Person objects to the Hadoop table
            using (var client = new WebHDFSClient(new Uri(clusterUri), userName, password))
            using (var writer = AvroContainer.CreateWriter<Person>(new HdfsFileLocation(tableLocation), new AvroSerializerSettings { Resolver = new AvroDataContractResolver(), UseCache = true }, Codec.Deflate))
            {
                foreach (var person in people)
                {
                    writer.Write(person);
                }
            }

            Console.WriteLine("Data has been written to the Hadoop table.");
        }
    }

    class Person
    {
        public string Name { get; set; }
        public int Age { get; set; }
    }
}

This code defines the Hadoop cluster credentials and endpoint, as well as the location and Avro schema for the table. It creates a list of Person objects to write to the table and uses the AvroContainer and WebHDFSClient classes from the Microsoft.Hadoop.Avro and Microsoft.Hadoop.WebHDFS.Adapters libraries, respectively, to write the data to the table. Note that you may need to add references to these libraries in your project before you can use them.

Pamai Tech
Turning ideas into Reality

Products

Office Add-in

Enterprise Solutions

Cloud Consulting

UI UX Design

Data Transformation

Services

FAQ's

Privacy Policy

Terms & Condition

Team

Contact Us

Company

About Us

Services

Features

Our Pricing

Latest News

© 2025 Pamai Tech