ANSWER :

1.1 Socket:

Socket

is bidirectional data path to a remote system.

A

socket is much like a file, except that a

single socket provides a two-way connection between two programs.

We can both

read from and write to the same socket. If you write something to a socket, it is sent to the application at the other end of the socket. If you read from the socket, you are given the data which the other application has sent.

1.2 Program to retrieve the data from a webserver:

Consider a simple program to retrieve the data from a web page. To understand the program given below, one should know the meaning of terminologies used there.

i.

AF_INET

is an address family (IP) that is used to designate the type of addresses

that your socket can communicate with. When you create a socket, you have to specify its address family, and then you can use only addresses of that type with the socket.

ii.

SOCK_STREAM

is a constant indicating the type of socket (TCP). It works as a file

stream and is most reliable over the network.

iii.

Port

is a logical end-point. Port 80 is one of the most commonly used

port

numbers

in the Transmission Control Protocol (TCP) suite.

iv. The command to retrieve the data must use CRLF(Carriage Return Line Feed) line endings, and it must end in \r\n\r\n (line break in protocol specification).

v.

encode()

method applied on strings will return bytes-representation of the string.

Instead of encode() method, one can attach a character b at the beginning of the string for the same effect.

vi.

decode()

method returns a string decoded from the given bytes.

A socket connection between the user program and the webpage is shown in Figure 1.1

Figure 1.1 A Socket Connection

Now, observe the following program -

When we run above program, we will get some information related to web-server of the website which we are trying to scrape. Then, we will get the data written in that web -page. In this program, we are extracting 167 bytes of data at a time. (One can use one's convenient number here). The extracted data is decoded and printed. When the length of data becomes less than one (that is, no more data left out on the web page), the loop is terminated.

ANSWER :

eXtensible Markup Language (XML) is best suited for exchanging document-style data.

XML looks very similar to HTML, but XML is more structured than HTML. Here is a sample of an XML document:

<person>

<name>Chuck</name>

<phone type="intl">

+1 734 303 4456

</phone>

<email hide="yes"/>

</person>

Often it is helpful to think of an XML document as a tree structure where there is a top tag person and other tags such as phone are drawn as children of their parent nodes. Figure 2.1 is the tree structure for above given XML code.

Fig 2.1 A Tree Representation of XML

2.2

Parsing XML

Python provides library xml.etree.ElementTree to parse the data from XML files. One has to provide XML code as a string to built-in method fromstring() of ElementTree class. ElementTree acts as a parser and provides a set of relevant methods to extract the data. Hence, the programmer need not know the rules and the format of XML document syntax. The fromstring()method will convert XML code into a tree-structure of XML nodes. When the XML is in a tree format, Python provides several methods to extract data from XML. Consider the following program.

In the above example, fromstring() is used to convert XML code into a tree. The find() method searches XML tree and retrieves a node that matches the specified tag. The get() method retrieves the value associated with the specified attribute of that tag. Each node can have some text, some attributes (like hide), and some "child" nodes. Each node can be the parent for a tree of nodes.

ANSWER :

3.1 Definition of Cursor:

A cursor() is like a file handle that we can use to perform operations on the data stored in the database. Calling cursor() is very similar conceptually to calling open() when dealing with text files. Hence, once we get a cursor, we can execute the commands on the contents of database using execute() method.

3.2 connect, execute and close command of databases:

Connect -> is used to connect a database with a name.

Execute -> is a command used to manipulate or access the mainly select create update drop.

Close -> is used to close the connection made by the connect command.

Fig 3.1 A Database Cursor

In the above program, we are inserting first record with the SQL command -

"INSERT INTO Tracks (title, plays) VALUES('Thunderstruck', 20)"

Note that, execute() requires SQL command to be in string format. But, if the value to be store in the table is also a string (TEXT type), then there may be a conflict of string representation using quotes. Hence, in this example, the entire SQL is mentioned within double-quotes and the value to be inserted in single quotes. If we would like to use either single quote or double quote everywhere, then we need to use escape-sequences like \' or \".

While inserting second row in a table, SQL statement is used with a little different syntax -

"INSERT INTO Tracks (title, plays) VALUES (?, ?)",('My Way', 15)

Here, the question mark acts as a place-holder for particular value. This type of syntax is useful when we would like to pass user-input values into database table.

After inserting two rows, we must use commit() method to store the inserted records permanently on the database table. If this method is not applied, then the insertion (or any other statement execution) will be temporary and will affect only the current run of the program.

Later, we use SELECT command to retrieve the data from the table and then use for-loop to display all records. When data is retrieved from database using SELECT command, the cursor object gets those data as a list of records. Hence, we can use for-loop on the cursor object. Finally, we have used a DELETE command to delete all the records WHERE plays is less than 100.

ANSWER :

Python provides simpler way of webpage retrieval using the library urllib. Here, webpage is treated like a file. urllib handles all of the HTTP protocol and header details. Following is the code equivalent to the program

import urllib.request

fhand = urllib.request.urlopen('http://data.pr4e.org/romeo.txt')

for line in fhand:

print(line.decode().strip())

Once the web page has been opened with urllib.urlopen, we can treat it like a file and read through it using a for-loop. When the program runs, we only see the output of the contents of the file. The headers are still sent, but the urllib code consumes the headers and only returns the data to us.

Following is the program to retrieve the data from the file romeo.txt which is residing at www.data.pr4e.org, and then to count number of words in it.

ANSWER :

5.1 JavaScript Object Notation (JSON)

The JSON format was inspired by the object and array format used in the JavaScript language. But since Python was invented before JavaScript, Python's syntax for dictionaries and lists influenced the syntax of JSON. So the format of JSON is a combination of Python lists and dictionaries. Following is the JSON encoding that is roughly equivalent to the XML code (the string data) given in the program of Question 2.

{

"name" : "Chuck",

"phone": {"type" : "intl", "number" : "+1 734 303 4456"},

"email": {"hide" : "yes"}

}

Observe the differences between XML code and JSON code:

i. In XML, we can add attributes like "intl" to the "phone" tag. In JSON, we simply have key-value pairs.

ii. XML uses tag "person", which is replaced by a set of outer curly braces in JSON.

In general, JSON structures are simpler than XML because JSON has fewer capabilities than XML. But JSON has the advantage that it maps directly to some combination of dictionaries and lists. And since nearly all programming languages have something equivalent to Python's dictionaries and lists, JSON is a very natural format to have two compatible programs exchange data. JSON is quickly becoming the format of choice for nearly all data exchange between applications because of its relative simplicity compared to XML.

5.2 Parsing JSON

Python provides a module json to parse the data in JSON pages. Consider the following program which uses JSON equivalent of XML string written in Section 5.2.3. Note that, the JSON string has to embed a list of dictionaries.

Here, the string data contains a list of users, where each user is a key-value pair. The method loads() in the json module converts the string into a list of dictionaries. Now onwards, we don't need anything from json, because the parsed data is available in Python native structures. Using a for-loop, we can iterate through the list of dictionaries and extract every element (in the form of key-value pair) as if it is a dictionary object. That is, we use index operator (a pair of square brackets) to extract value for a particular key.

NOTE:

Current IT industry trend is to use JSON for web services rather than XML.

Because, JSON is simpler than XML and it directly maps to native data structures we already have in the programming languages. This makes parsing and data extraction simpler compared to XML. But XML is more self descriptive than JSON and so there are some applications where XML retains an advantage. For example, most word processors store documents internally using XML rather than JSON.