Have Geek Brain

Calling a web page from Linux C

I need to read all HTML text from a url, for example http://localhost/index.html

into a string in C.

I know that if I put telnet -> telnet www.google.com 80 Get webpage....

it on it will return all the html.

How can I do this in a linux environment using C?

+2


a source to share


5 answers


I would suggest using several libraries that are commonly available for most Linux distributions:

libcurl and libxml2



libcurl provides a full set of http functions and libxml2 provides a module for html parsing called HTMLParser

Hope you point in the right direction

+6


a source


Below is some example code (i.e. not much error checking and I haven't tried to compile it) to get started, but use http://www.tenouk.com/cnlinuxsockettutorials.html to learn socket programming. Have a look at gethostbyname if you need to translate a hostname (like google.com) to an IP address. Also, you may need to do some work of parsing the length of the content from the HTTP response, and then make sure you keep calling recv until you get all the bytes.



#include <netinet/in.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <string.h>
#include <stdlib.h>

void getWebpage(char *buffer, int bufsize, char *ipaddress)
{
    int sockfd;
    struct sockaddr_in destAddr;

    if((sockfd = socket(PF_INET, SOCK_STREAM, 0)) == -1){
        fprintf(stderr, "Error opening client socket\n");
        close(sockfd);
        return;
    }

    destAddr.sin_family = PF_INET;
    destAddr.sin_port = htons(80); // HTTP port is 80
    destAddr.sin_addr.s_addr = inet_addr(ipaddress); // Get int representation of IP
    memset(&(destAddr.sin_zero), 0, 8);

    if(connect(sockfd, (struct sockaddr *)&destAddr, sizeof(struct sockaddr)) == -1){
        fprintf(stderr, "Error with client connecting to server\n");
        close(sockfd);
        return;
    }

    // Send http request
    char *httprequest = "GET / HTTP/1.0";
    send(sockfd, httprequest, strlen(httprequest), 0);
    recv(sockfd, buffer, bufsize, 0);

    // Now buffer has the HTTP response which includes the webpage. You can either
    // trim off the HTTP header, or just leave it in depending on what you are doing
    // with the page
}
+3


a source


You are using sockets, poll the web server with HTTP (where you have http: //localhost/index.html ") and then parse the data you received.

Useful if you are new to socket programming: http://beej.us/guide/bgnet/

+1


a source


if you really don't feel like messing with sockets, you can always create a named temp file, unblock the process, and execute execvp () to run wget -0 and then read the input of that temp file.

while this would be a rather lame and inefficient way of doing things, it would mean that you would not need to communicate with TCP and send HTTP requests.

+1


a source


Assuming you know how to read a file into a string, I would try

const char *url_contents(const char *url) {
  // create w3m command and pass it to popen()
  int bufsize = strlen(url) + 100;
  char *buf = malloc(bufsize);
  snprintf(buf, bufsize, "w3m -dump_source '%s'");

  // get a file handle, read all the html from it, close, and return
  FILE *html = popen(buf, "r");
  const char *s = read_file_into_string(html); // you write this function
  fclose(html);
  return s;
}

You are enacting the process, but it’s much easier to let the w3m

hard climb do.

0


a source






More articles:


All Articles