Thursday, September 12, 2019

Computer Science Info-sec website protection with sql -injection

The Target Intranet

This appeared to be an entirely custom application, and we had no prior knowledge of the application nor access to the source code: this was a "blind" attack. A bit of poking showed that this server ran Microsoft's IIS 6 along with ASP.NET, and this suggested that the database was Microsoft's SQL server: we believe that these techniques can apply to nearly any web application backed by any SQL server.
The login page had a traditional username-and-password form, but also an email-me-my-password link; the latter proved to be the downfall of the whole system.
When entering an email address, the system presumably looked in the user database for that email address, and mailed something to that address. Since my email address is not found, it wasn't going to send me anything.
So the first test in any SQL-ish form is to enter a single quote as part of the data: the intention is to see if they construct an SQL string literally without sanitizing. When submitting the form with a quote in the email address, we get a 500 error (server failure), and this suggests that the "broken" input is actually being parsed literally. Bingo.
We speculate that the underlying SQL code looks something like this:
SELECT fieldlist
FROM table
WHERE field = '$EMAIL';
Here, $EMAIL is the address submitted on the form by the user, and the larger query provides the quotation marks that set it off as a literal string. We don't know the specific names of the fields or table involved, but we do know their nature, and we'll make some good guesses later.
When we enter steve@unixwiz.net' - note the closing quote mark - this yields constructed SQL:
SELECT fieldlist
FROM table
WHERE field = 'steve@unixwiz.net'';
when this is executed, the SQL parser find the extra quote mark and aborts with a syntax error. How this manifests itself to the user depends on the application's internal error-recovery procedures, but it's usually different from "email address is unknown". This error response is a dead giveaway that user input is not being sanitized properly and that the application is ripe for exploitation.
Since the data we're filling in appears to be in the WHERE clause, let's change the nature of that clause in an SQL legal way and see what happens. By entering anything' OR 'x'='x, the resulting SQL is:
SELECT fieldlist
FROM table
WHERE field = 'anything' OR 'x'='x';
Because the application is not really thinking about the query - merely constructing a string - our use of quotes has turned a single-component WHERE clause into a two-component one, and the 'x'='x' clause is guaranteed to be true no matter what the first clause is (there is a better approach for this "always true" part that we'll touch on later).
But unlike the "real" query, which should return only a single item each time, this version will essentially return every item in the members database. The only way to find out what the application will do in this circumstance is to try it. Doing so, we were greeted with:

Your login information has been mailed to random.person@example.com.
Our best guess is that it's the first record returned by the query, effectively an entry taken at random. This person really did get this forgotten-password link via email, which will probably come as surprise to him and may raise warning flags somewhere.
We now know that we're able to manipulate the query to our own ends, though we still don't know much about the parts of it we cannot see. But we have observed three different responses to our various inputs:
  • "Your login information has been mailed to email"
  • "We don't recognize your email address"
  • Server error
The first two are responses to well-formed SQL, while the latter is for bad SQL: this distinction will be very useful when trying to guess the structure of the query.

Schema field mapping

The first steps are to guess some field names: we're reasonably sure that the query includes "email address" and "password", and there may be things like "US Mail address" or "userid" or "phone number". We'd dearly love to perform a SHOW TABLE, but in addition to not knowing the name of the table, there is no obvious vehicle to get the output of this command routed to us.
So we'll do it in steps. In each case, we'll show the whole query as we know it, with our own snippets shown specially. We know that the tail end of the query is a comparison with the email address, so let's guess email as the name of the field:
SELECT fieldlist
FROM table
WHERE field = 'x' AND email IS NULL; --';
The intent is to use a proposed field name (email) in the constructed query and find out if the SQL is valid or not. We don't care about matching the email address (which is why we use a dummy 'x'), and the -- marks the start of an SQL comment. This is an effective way to "consume" the final quote provided by application and not worry about matching them.
If we get a server error, it means our SQL is malformed and a syntax error was thrown: it's most likely due to a bad field name. If we get any kind of valid response, we guessed the name correctly. This is the case whether we get the "email unknown" or "password was sent" response.
Note, however, that we use the AND conjunction instead of OR: this is intentional. In the SQL schema mapping phase, we're not really concerned with guessing any particular email addresses, and we do not want random users inundated with "here is your password" emails from the application - this will surely raise suspicions to no good purpose. By using the AND conjunction with an email address that couldn't ever be valid, we're sure that the query will always return zero rows and never generate a password-reminder email.
Submitting the above snippet indeed gave us the "email address unknown" response, so now we know that the email address is stored in a field email. If this hadn't worked, we'd have tried email_address or mail or the like. This process will involve quite a lot of guessing.
Next we'll guess some other obvious names: password, user ID, name, and the like. These are all done one at a time, and anything other than "server failure" means we guessed the name correctly.
SELECT fieldlist
FROM table
WHERE email = 'x' AND userid IS NULL; --';
As a result of this process, we found several valid field names:
  • email
  • passwd
  • login_id
  • full_name
There are certainly more (and a good source of clues is the names of the fields on forms), but a bit of digging did not discover any. But we still don't know the name of the table that these fields are found in - how to find out?

Finding the table name

The application's built-in query already has the table name built into it, but we don't know what that name is: there are several approaches for finding that (and other) table names. The one we took was to rely on a subselect.
A standalone query of
SELECT COUNT(*) FROM tabname
Returns the number of records in that table, and of course fails if the table name is unknown. We can build this into our string to probe for the table name:
SELECT email, passwd, login_id, full_name
FROM table
WHERE email = 'x' AND 1=(SELECT COUNT(*) FROM tabname); --';
We don't care how many records are there, of course, only whether the table name is valid or not. By iterating over several guesses, we eventually determined that members was a valid table in the database. But is it the table used in this query? For that we need yet another test using table.field notation: it only works for tables that are actually part of this query, not merely that the table exists.
SELECT email, passwd, login_id, full_name
FROM members
WHERE email = 'x' AND members.email IS NULL; --';
When this returned "Email unknown", it confirmed that our SQL was well formed and that we had properly guessed the table name. This will be important later, but we instead took a different approach in the interim.

Finding some users

At this point we have a partial idea of the structure of the members table, but we only know of one username: the random member who got our initial "Here is your password" email. Recall that we never received the message itself, only the address it was sent to. We'd like to get some more names to work with, preferably those likely to have access to more data.
The first place to start, of course, is the company's website to find who is who: the "About us" or "Contact" pages often list who's running the place. Many of these contain email addresses, but even those that don't list them can give us some clues which allow us to find them with our tool.
The idea is to submit a query that uses the LIKE clause, allowing us to do partial matches of names or email addresses in the database, each time triggering the "We sent your password" message and email. Warning: though this reveals an email address each time we run it, it also actually sends that email, which may raise suspicions. This suggests that we take it easy.
We can do the query on email name or full name (or presumably other information), each time putting in the % wildcards that LIKE supports:
SELECT email, passwd, login_id, full_name
FROM members
WHERE email = 'x' OR full_name LIKE '%Bob%';
Keep in mind that even though there may be more than one "Bob", we only get to see one of them: this suggests refining our LIKE clause narrowly.
Ultimately, we may only need one valid email address to leverage our way in.

Brute-force password guessing

One can certainly attempt brute-force guessing of passwords at the main login page, but many systems make an effort to detect or even prevent this. There could be logfiles, account lockouts, or other devices that would substantially impede our efforts, but because of the non-sanitized inputs, we have another avenue that is much less likely to be so protected.
We'll instead do actual password testing in our snippet by including the email name and password directly. In our example, we'll use our victim, bob@example.com and try multiple passwords.
SELECT email, passwd, login_id, full_name
FROM members
WHERE email = 'bob@example.com' AND passwd = 'hello123';
This is clearly well-formed SQL, so we don't expect to see any server errors, and we'll know we found the password when we receive the "your password has been mailed to you" message. Our mark has now been tipped off, but we do have his password.
This procedure can be automated with scripting in perl, and though we were in the process of creating this script, we ended up going down another road before actually trying it.

The database isn't readonly

So far, we have done nothing but query the database, and even though a SELECT is readonly, that doesn't mean that SQL is. SQL uses the semicolon for statement termination, and if the input is not sanitized properly, there may be nothing that prevents us from stringing our own unrelated command at the end of the query.
The most drastic example is:
SELECT email, passwd, login_id, full_name
FROM members
WHERE email = 'x'; DROP TABLE members; --';  -- Boom!
The first part provides a dummy email address -- 'x' -- and we don't care what this query returns: we're just getting it out of the way so we can introduce an unrelated SQL command. This one attempts to drop (delete) the entire members table, which really doesn't seem too sporting.
This shows that not only can we run separate SQL commands, but we can also modify the database. This is promising.

Adding a new member

Given that we know the partial structure of the members table, it seems like a plausible approach to attempt adding a new record to that table: if this works, we'll simply be able to login directly with our newly-inserted credentials.
This, not surprisingly, takes a bit more SQL, and we've wrapped it over several lines for ease of presentation, but our part is still one contiguous string:
SELECT email, passwd, login_id, full_name
FROM members
WHERE email = 'x';
INSERT INTO members ('email','passwd','login_id','full_name') 
VALUES ('steve@unixwiz.net','hello','steve','Steve Friedl');--';
Even if we have actually gotten our field and table names right, several things could get in our way of a successful attack:
  1. We might not have enough room in the web form to enter this much text directly (though this can be worked around via scripting, it's much less convenient).
  2. The web application user might not have INSERT permission on the members table.
  3. There are undoubtedly other fields in the members table, and some may require initial values, causing the INSERT to fail.
  4. Even if we manage to insert a new record, the application itself might not behave well due to the auto-inserted NULL fields that we didn't provide values for.
  5. A valid "member" might require not only a record in the members table, but associated information in other tables (say, "accessrights"), so adding to one table alone might not be sufficient.
In the case at hand, we hit a roadblock on either #4 or #5 - we can't really be sure -- because when going to the main login page and entering in the above username + password, a server error was returned. This suggests that fields we did not populate were vital, but nevertheless not handled properly.
A possible approach here is attempting to guess the other fields, but this promises to be a long and laborious process: though we may be able to guess other "obvious" fields, it's very hard to imagine the bigger-picture organization of this application.
We ended up going down a different road.

Mail me a password

We then realized that though we are not able to add a new record to the members database, we can modify an existing one, and this proved to be the approach that gained us entry.
From a previous step, we knew that bob@example.com had an account on the system, and we used our SQL injection to update his database record with our email address:
SELECT email, passwd, login_id, full_name
FROM members
WHERE email = 'x';
UPDATE members
SET email = 'steve@unixwiz.net'
WHERE email = 'bob@example.com';
After running this, we of course received the "we didn't know your email address", but this was expected due to the dummy email address provided. The UPDATE wouldn't have registered with the application, so it executed quietly.
We then used the regular "I lost my password" link - with the updated email address - and a minute later received this email:
Now it was now just a matter of following the standard login process to access the system as a high-ranked MIS staffer, and this was far superior to a perhaps-limited user that we might have created with our INSERT approach.
We found the intranet site to be quite comprehensive, and it included - among other things - a list of all the users. It's a fair bet that many Intranet sites also have accounts on the corporate Windows network, and perhaps some of them have used the same password in both places. Since it's clear that we have an easy way to retrieve any Intranet password, and since we had located an open PPTP VPN port on the corporate firewall, it should be straightforward to attempt this kind of access.
We had done a spot check on a few accounts without success, and we can't really know whether it's "bad password" or "the Intranet account name differs from the Windows account name". But we think that automated tools could make some of this easier.

Other Approaches

In this particular engagement, we obtained enough access that we did not feel the need to do much more, but other steps could have been taken. We'll touch on the ones that we can think of now, though we are quite certain that this is not comprehensive.
We are also aware that not all approaches work with all databases, and we can touch on some of them here.
Use xp_cmdshell
Microsoft's SQL Server supports a stored procedure xp_cmdshell that permits what amounts to arbitrary command execution, and if this is permitted to the web user, complete compromise of the webserver is inevitable.
What we had done so far was limited to the web application and the underlying database, but if we can run commands, the webserver itself cannot help but be compromised. Access to xp_cmdshell is usually limited to administrative accounts, but it's possible to grant it to lesser users.
Map out more database structure
Though this particular application provided such a rich post-login environment that it didn't really seem necessary to dig further, in other more limited environments this may not have been sufficient.
Being able to systematically map out the available schema, including tables and their field structure, can't help but provide more avenues for compromise of the application.
One could probably gather more hints about the structure from other aspects of the website (e.g., is there a "leave a comment" page? Are there "support forums"?). Clearly, this is highly dependent on the application and it relies very much on making good guesses.

Mitigations

We believe that web application developers often simply do not think about "surprise inputs", but security people do (including the bad guys), so there are three broad approaches that can be applied here.
Sanitize the input
It's absolutely vital to sanitize user inputs to insure that they do not contain dangerous codes, whether to the SQL server or to HTML itself. One's first idea is to strip out "bad stuff", such as quotes or semicolons or escapes, but this is a misguided attempt. Though it's easy to point out some dangerous characters, it's harder to point to all of them.
The language of the web is full of special characters and strange markup (including alternate ways of representing the same characters), and efforts to authoritatively identify all "bad stuff" are unlikely to be successful.
Instead, rather than "remove known bad data", it's better to "remove everything but known good data": this distinction is crucial. Since - in our example - an email address can contain only these characters:
abcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789
@.-_+
There is really no benefit in allowing characters that could not be valid, and rejecting them early - presumably with an error message - not only helps forestall SQL Injection, but also catches mere typos early rather than stores them into the database.
Sidebar on email addresses
It's important to note here that email addresses in particular are troublesome to validate programmatically, because everybody seems to have his own idea about what makes one "valid", and it's a shame to exclude a good email address because it contains a character you didn't think about. The only real authority is RFC 2822 (which encompasses the more familiar RFC822), and it includes a fairly expansive definition of what's allowed. The truly pedantic may well wish to accept email addresses with ampersands and asterisks (among other things) as valid, but others - including this author - are satisfied with a reasonable subset that includes "most" email addresses. Those taking a more restrictive approach ought to be fully aware of the consequences of excluding these addresses, especially considering that better techniques (prepare/execute, stored procedures) obviate the security concerns which those "odd" characters present.
Be aware that "sanitizing the input" doesn't mean merely "remove the quotes", because even "regular" characters can be troublesome. In an example where an integer ID value is being compared against the user input (say, a numeric PIN):
SELECT fieldlist
FROM table
WHERE id = 23 OR 1=1;  -- Boom! Always matches!
In practice, however, this approach is highly limited because there are so few fields for which it's possible to outright exclude many of the dangerous characters. For "dates" or "email addresses" or "integers" it may have merit, but for any kind of real application, one simply cannot avoid the other mitigations.
Escape/Quotesafe the input
Even if one might be able to sanitize a phone number or email address, one cannot take this approach with a "name" field lest one wishes to exclude the likes of Bill O'Reilly from one's application: a quote is simply a valid character for this field.
One includes an actual single quote in an SQL string by putting two of them together, so this suggests the obvious - but wrong! - technique of preprocessing every string to replicate the single quotes:
SELECT fieldlist
FROM customers
WHERE name = 'Bill O''Reilly';  -- works OK
However, this naïve approach can be beaten because most databases support other string escape mechanisms. MySQL, for instance, also permits \' to escape a quote, so after input of \'; DROP TABLE users; -- is "protected" by doubling the quotes, we get:
SELECT fieldlist
FROM customers
WHERE name = '\''; DROP TABLE users; --';  -- Boom!
The expression '\'' is a complete string (containing just one single quote), and the usual SQL shenanigans follow. It doesn't stop with backslashes either: there is Unicode, other encodings, and parsing oddities all hiding in the weeds to trip up the application designer.
Getting quotes right is notoriously difficult, which is why many database interface languages provide a function that does it for you. When the same internal code is used for "string quoting" and "string parsing", it's much more likely that the process will be done properly and safely.
Some examples are the MySQL function mysql_real_escape_string() and perl DBD method $dbh->quote($value).
These methods must be used.
Use bound parameters (the PREPARE statement)
Though quotesafing is a good mechanism, we're still in the area of "considering user input as SQL", and a much better approach exists: bound parameters, which are supported by essentially all database programming interfaces. In this technique, an SQL statement string is created with placeholders - a question mark for each parameter - and it's compiled ("prepared", in SQL parlance) into an internal form.
Later, this prepared query is "executed" with a list of parameters:
Example in perl
$sth = $dbh->prepare("SELECT email, userid FROM members WHERE email = ?;");

$sth->execute($email);
Thanks to Stefan Wagner, this demonstrates bound parameters in Java:
Insecure version
Statement s = connection.createStatement();
ResultSet rs = s.executeQuery("SELECT email FROM member WHERE name = "
+ formField); // *boom*
Secure version
PreparedStatement ps = connection.prepareStatement(
"SELECT email FROM member WHERE name = ?");
ps.setString(1, formField);
ResultSet rs = ps.executeQuery();
Here, $email is the data obtained from the user's form, and it is passed as positional parameter #1 (the first question mark), and at no point do the contents of this variable have anything to do with SQL statement parsing. Quotes, semicolons, backslashes, SQL comment notation - none of this has any impact, because it's "just data". There simply is nothing to subvert, so the application is be largely immune to SQL injection attacks.
There also may be some performance benefits if this prepared query is reused multiple times (it only has to be parsed once), but this is minor compared to the enormous security benefits. This is probably the single most important step one can take to secure a web application.
Limit database permissions and segregate users
In the case at hand, we observed just two interactions that are made not in the context of a logged-in user: "log in" and "send me password". The web application ought to use a database connection with the most limited rights possible: query-only access to the members table, and no access to any other table.
The effect here is that even a "successful" SQL injection attack is going to have much more limited success. Here, we'd not have been able to do the UPDATE request that ultimately granted us access, so we'd have had to resort to other avenues.
Once the web application determined that a set of valid credentials had been passed via the login form, it would then switch that session to a database connection with more rights.
It should go almost without saying that sa rights should never be used for any web-based application.
Use stored procedures for database access
When the database server supports them, use stored procedures for performing access on the application's behalf, which can eliminate SQL entirely (assuming the stored procedures themselves are written properly).
By encapsulating the rules for a certain action - query, update, delete, etc. - into a single procedure, it can be tested and documented on a standalone basis and business rules enforced (for instance, the "add new order" procedure might reject that order if the customer were over his credit limit).
For simple queries this might be only a minor benefit, but as the operations become more complicated (or are used in more than one place), having a single definition for the operation means it's going to be more robust and easier to maintain.
Note: it's always possible to write a stored procedure that itself constructs a query dynamically: this provides no protection against SQL Injection - it's only proper binding with prepare/execute or direct SQL statements with bound variables that provide this protection.
Isolate the webserver
Even having taken all these mitigation steps, it's nevertheless still possible to miss something and leave the server open to compromise. One ought to design the network infrastructure to assume that the bad guy will have full administrator access to the machine, and then attempt to limit how that can be leveraged to compromise other things.
For instance, putting the machine in a DMZ with extremely limited pinholes "inside" the network means that even getting complete control of the webserver doesn't automatically grant full access to everything else. This won't stop everything, of course, but it makes it a lot harder.
Configure error reporting
The default error reporting for some frameworks includes developer debugging information, and this cannot be shown to outside users. Imagine how much easier a time it makes for an attacker if the full query is shown, pointing to the syntax error involved.
This information is useful to developers, but it should be restricted - if possible - to just internal users.
Note that not all databases are configured the same way, and not all even support the same dialect of SQL (the "S" stands for "Structured", not "Standard"). For instance, most versions of MySQL do not support subselects, nor do they usually allow multiple statements: these are substantially complicating factors when attempting to penetrate a network.

We'd like to emphasize that though we chose the "Forgotten password" link to attack in this particular case, it wasn't really because this particular web application feature is dangerous. It was simply one of several available features that might have been vulnerable, and it would be a mistake to focus on the "Forgotten password" aspect of the presentation.
This Tech Tip has not been intended to provide comprehensive coverage on SQL injection, or even a tutorial: it merely documents the process that evolved over several hours during a contracted engagement. We've seen other papers on SQL injection discuss the technical background, but still only provide the "money shot" that ultimately gained them access.
But that final statement required background knowledge to pull off, and the process of gathering that information has merit too. One doesn't always have access to source code for an application, and the ability to attack a custom application blindly has some value.

Monday, February 6, 2012

Vim as a Universal IDE {Part 2 (Ruby/Rails )}

when a Newbie starts coding with ruby , it seems hard to switch between files
and running scripts .It seem a lot of time consuming with coding
ruby . Many people ask on irc and groups daily which is best IDE for Ruby.Or
some those used Textmate,Eclispe ,etc.. are also facing time consumption
while write some code! SO ! SO! SO! Now Solution is VIM!

I already Explained basic Installation and working on vim in my previous post
those who are new to vim can visit Part 1 for basic editing and installation
,configuration of vim for better use

I explain here some useful plug-in and working,configuration of those .This
post help ! you to shape code perfect with vim and ruby.
Comments Welcome !!

1) For Enhanced Syntax Highlighting & Navigation(rails.vim)
 automatically detects buffers ,it effect only rails script not ruby scripts
.Easy navigation and interface to script ,integration with other plugins.such
NERDtree.vim with :Rtree

install details
extract the zip to ~/.vim

Download page is here http://www.vim.org/scripts/script.php?script_id=1567

2)Organization of project like (IDE/Buffer)
this help you launch a .*.jpg files in a viewer. I have also set
up viewers for PDF (acroread) and HTML files (mozilla) for my own use.

install details
untar the file in ~/.vim
for untar use this command :- tar -xzvf filename
download page link is here  http://www.vim.org/scripts/script.php?script_id=69

3)Ruby filetype and indentation
:-

a)for Ruby indentation :download indentation script and extract(gunzip
filename) gzip  in ~/.vim/indent
download link is here  http://www.vim.org/scripts/script.php?script_id=2742
along with this you need install a plugin Indentanything

install details for that is here    
cd ~/.vim
tar zxvf IndentAnything.tar.gz
download page link is here http://www.vim.org/scripts/script.php?script_id=1839

b)matchit.zip
: extended % matching for HTML, LaTeX, and many other
languages The matchit.vim script allows you to configure % to match more    than just single characters.  You can match words and even regular expressions.
Also, matching treats strings and comments (as recognized by the
syntax highlighting mechanism) intelligently.
The default ftplugins include settings for several languages:
Ada, ASP with VBS, Csh, DTD, Essbase, Fortran, HTML, JSP
(same as HTML), LaTeX, Lua, Pascal, SGML, Shell, Tcsh, Vim, XML.
(I no longer keep track, so there may be others.)

install details
extract zip in ~/.vim

download page link is here http://www.vim.org/scripts/script.php?script_id=39
your .vimrc config must have below one
filetype plugin on

c)ruby-matchit:- Map '%' to jump from one keyword to its corresponding
'end' in Ruby files. Inspired by 'matchit.vim' that comes with Vim

install details

Put the file in ~/.vim/ftplugin/ruby/ directory
download page http://www.vim.org/scripts/script.php?script_id=290


4)Database excess ( dbext.vim) : Provides database access to many dbms such as: Mysql, PostgreSQL, Ingres, Oracle, Oracle Rdb (VMS), Sybase Adaptive Server Anywhere, Sybase Adaptive Server Enterprise,iMicrosoft SQL Server, DB2, Interbase and SQLite and ODBC are supported. 
 install deatils
 extract zip in ~/.vim
 download page is here http://www.vim.org/scripts/script.php?script_id=356



 5) Haml and Sass syntax, indenting, and ftplugin :-

 Runtime files for Haml and Sass.
 install details
 Extract in ~/.vim or ~\vimfiles.
 download page is here http://www.vim.org/scripts/script.php?script_id=1433




6) Ruby-test
:- Rubytest.vim will help you to run ruby test (including        vanillatest, rspec, shoulda etc.) in vim.

Installation
Unzip downloaded file and copy rubytest.vim to your ~/.vim/plugin directory.
download page for the this http://www.vim.org/scripts/script.php?script_id=2612

Your .vimrc config you need to do is below
let g:rubytest_cmd_test = "ruby %p"
let g:rubytest_cmd_testcase = "ruby %p -n '/%c/'"
let g:rubytest_cmd_spec = "spec -f specdoc %p"
let g:rubytest_cmd_example = "spec -f specdoc %p -e '%c'"            
let g:rubytest_cmd_feature = "cucumber %p"
let g:rubytest_cmd_story = "cucumber %p -n '%c'"

Default keybind to change

map <Leader>\ <Plug>RubyTestRun " change from <Leader>t to <Leader>\
map <Leader>] <Plug>RubyFileRun " change from <Leader>T to <Leader>]
map <Leader>/ <Plug>RubyTestRunLast " change from <Leader>l to <Leader>/
     


7) cucumber runtime :- description
This provides syntax highlighting, indenting, and a filetype plugin for
Cucumber, the acceptance testing framework

install details
Extract in ~/.vim

download page   http://www.vim.org/scripts/script.php?script_id=2973


8) Endwise.vim
: Wisely add :-
  This is a simple plugin that helps to end certain structures automatically.In Ruby, this means adding "end" after "if", "do", "def" and several other
  keywords. In Vim Script, this amounts to appropriately adding "endfunction","endif", etc

  install deatils
  install in ~/.vim/plugin
  download page is here http://www.vim.org/scripts/script.php?script_id=2386



  9) Genutils : General utility functions :-

  This script provides functions that are mostly useful to script developers,but some of the functions can be easily converted to good utilities.
  install details
  extarct in ~/.vim 
  download page http://www.vim.org/scripts/script.php?script_id=197


 now config your .vimrc
command! -nargs=0 -range=% SortByLength <line1>,<line2>call QSort(
        \ 'CmpByLineLengthNname', 1)
command! -nargs=0 -range=% RSortByLength <line1>,<line2>call QSort(
        \ 'CmpByLineLengthNname', -1)
command! -nargs=0 -range=% SortJavaImports <line1>,<line2>callQSort(
        \ 'CmpJavaImports', 1)


u might like the following mappings to adjust spacing:
nnoremap <silent> <C-Space> :call ShiftWordInSpace(1)<CR>
nnoremap <silent> <C-BS> :call ShiftWordInSpace(-1)<CR>
nnoremap <silent> \cw :call CenterWordInSpace()<CR>
nnoremap <silent> \va :call
AlignWordWithWordInPreviousLine()<CR>

The :find command is very useful to search for a file in path, but it
doesn't support file completion. Add the following command in your vimrc
to add this functionality
nd! -nargs=1 -bang -complete=customlist,genutils#UserFileComplete2
            \ FindInPath :find<bang> <args>

If you are running commands that generate multiple pages of output, you
might find it useful to redirect the output to a new buffer. Put the
following command in your vimrc:
nd! -nargs=* -complete=command Redir
          \ :new | put! =GetVimCmdOutput('<args>') | setl bufhidden=wipe |
          \ setl nomodified

10) multvals.vim
: Array library that uses patterns as separators An array is
nothing but a string of multiple values separated by a
pattern.  The simplest example being Vim's multi-value variables such as
tags. You can use the MvAddElement() function to create an array.
However, there is nothing special about this function, you can as well
make up the string by simply concatenating elements with the chosen
pattern as a separator.

install in ~/.vim/plugin
download page is here http://www.vim.org/scripts/script.php?script_id=171

Some other plugin such as Nerd tree or git integration ,supertab ,code navigation are  used in previous post for python. Also textmate like completion and indentation also discuss in previous part .


Note:- Stay Tuned more parts of this Post coming soon for Ruby,C++,C,Closure,Prolog ,Haskell,Java,PHP,Perl,Shell, JavaScript ,HTML,CSS Coming soon (within few Days) .
Update of this post will done soon for Usage of plugin and more config Details .Also preparing image graphics for all post soon update all with how use effectively all these  plug-ins.


                                                                     THANKS FOR VISITING


LHS AS A SOURCE OF INFORMATION – AND A SOURCE OF INSPIRATION – I HOPE YOU’LL CHOOSE TO ACT RIGHT NOW.ENJOY KEEP LEARNING.

   

Saturday, February 4, 2012

VIM as an Universal IDE{Part 1 (Python/Django)}



Vim is My First and Last editor ,IDE, or you can say most of time i spend on it  while work on system .

I write this blog for newbies , those who new to both Linux and Programming.Also for professionals and passionate guys.Also give a look if you are using  different editor then Vim for any Programming language .I write everything that a newbie need to know from installation to configuration , from novice to advance
Any suggestions ! Comments Welcome !here

More about Vim 
Vim is worth millions and costs nothing. It's the best editor the world has
ever seen. Using the trio Vim/Perl/LaTeX I can dispose of most other software.

More To do :

1. Get VIM for Dos
2. Get VIM for Linux
3. Get Vim for Freebsd
4. Get Vim for Dec-Unix
5. Get VIM for Irix
6. Get VIM for Window
7. Get VIM for Mac
8. Laugh at non-VIM users



Basic Introduction :- I Know for Beginner's ,Newbie for Linux ,Vim Need to get in with this From a basic . Following Question can asked by a newbie to Vim
Note:- Please experts those know vim well or For geeks those know it well go below some (stuff is there ) 
   1) May you don't know install Vim on Your's OS(Linux,Mac,Unix etc)
   2) May be you don't know how to open Vim on your OS
   3) May Be You Don't know how install plugin for it
   4) May Be You Don't know how to close it

 Okay before answering you all guys , It work for a large range of Operating System's such as Linux,Mac,Window ,Unix , FreeBSD ..much more.It provide both Command line and GUI interface .


1)Installing Vim
     A)For Linux User's :-                

a) For Ubuntu/Debian user's :- sudo apt-get install vim            
b) For Fedora user's :-  yum install vim-X11 vim-common vim-enhanced vim-minimal
c) For Red-hat Linux user's :- up2date -i vim-X11 vim-common vim-enhanced vim-minimal
d) For Arch Linux user's:- pacman -S vim              
e) For Fedora user's:- emerge vim

B)For  Mac User's :- MacVim is the text editor for Mac OS X , download it form here http://code.google.com/p/macvim/

C)For  Window User's :-Go to  Vim Download page and click on “PC: MS-DOS and MS-Windows”. Click on the ‘gvim72.exe, which is a Self-installing executable.
(or) Download gvim72.exe directly.          
For All Others Visit download page on http://www.vim.org/download.php


2) Opening or Simple use of Vim
   You  can open a using command line :- use vi/vim  in start of any file name or any file you want open like below        
   $ vim hack.txt                  
   above use of vim with hack.txt file to open

Vim has two mode visual mode (GUI mode) and insert mode(command line mode).     To Switch in Between them press Esc on your keyboard and then Press i for Insert mode and v for visual mode on your keyboard .Esc play a important role for switch you any command . Or you can use Crtl+c ,that is an alternative of Esc .

Note : If not get now also ,then Visit you sure got all open close and simple to advance use link below    http://www.openvim.com/tutorial.html

3)  Vim have need to have two special area where all vim operate or contains  .Let me Explain
          1) create a ~/.vim directory
          $ mkdir ~/.vim            
          $ cd ~/.vim          
Note:- this Contain all your plugin installation.
       
          2) Now create file .vimrc under your bash or command terminal                  
           $vim .vimrc                  
Above will create and open .vimrc file in your bash and command terminal            

Note:- This contain information about your  configuration of vim and it's plugins

4)  Close it with press ESc on your Keyboard and then shift + : , then enter wq , wq! , q!            
 these all can use close the vim


Now Vim from Scratch to Novice (You can  say from level 1 to level 2 ):

Just visit and learn there the best online Tutorial  http://www.openvim.com/tutorial.html

Now move on Real topic Vim as Universal IDE(Integrated Development Environment ):-


 1) Vim as Python/Django IDE :-


 i) Now you need to use ~/.vim and .vimrc i as tell you earlier  Now first step set a better color-scheme wombat,download wombat.vim http://files.werx.dk/wombat.vim

Now under after downloading under ~/.vim/colors    
$ mkdir -p ~/.vim/colors          
$ cd ~/.vim/colors          
$ wget -O wombat.vim http://files.werx.dk/wombat.vim
$ vim
Now in vim press shift + : ,then enter color wombat like below          
$:color wombat
Now after perfect looking move on !!


ii) For Syntax Error and Formatting (Indent):-
This can used for indentation or formatting every language need formatting and indentation  of that language .This indentation script for python tries to match more closely what is suggested in PEP 8 (http://www.python.org/peps/pep-0008.html).  In particular, it handles continuation lines implied by open (parentheses), [brackets] and {braces} correctly and it indents multiline if/for/while statements differently.  Comments are welcome!
install details

Drop the script in your ~/.vim/indent directory.
$mkdir ~/.vim/indent $ cd ~/.vim/indent  
$ wget -O python.vim http://www.vim.org/scripts/download_script.php?src_id=4316

download page is here http://www.vim.org/scripts/script.php?script_id=974

Now config this in .vimrc    
$ vim .vimrc    
add below  to .vimrc
filetype plugin indent on



iii) Special highlighting for String  and String formatting (Python.vim) :-        

Enhanced version of the original (from vim6.1) python.vim for Python programming language.
The changes since the original python.vim are:
- changed strings highlighting;
- enhanced special symbols highlighting inside strings;
- enhanced numbers highlighting;
- added optional highlighting for %-formatting inside strings;
- added highlighting for some error conditions (wrong symbols in source file,  
mixing spaces and tabs, wrong number values,  
wrong %-formatting inside strings);
- added highlighting for magic comments: source code encoding
 and #! (executable) strings;  - added highlighting
for new exceptions and builtins introduced in python 2.3, 2.4 and 2.5;
- added highlighting for doctests;
- added highlighting for new @decorator syntax introduced in Python 2.4a2;
- added highlighting for trailing-space errors (triggered by new  option: python_highlight_space_errors);
- added highlighting for variable name errors;
- added highlighting for hex number errors;

install details
Place python.vim file in ~/.vim/syntax/ folder.

$mkdir ~/.vim/syntax
$cd ~/.vim/syntax
$wget -O python.vim http://www.vim.org/scripts/download_script.php?src_id=14268 or download page

download page is here http://www.vim.org/scripts/script.php?script_id=790          

Also put config in .vimrc is below
autocmd FileType python set complete+=k~/.vim/syntax/python.vim isk+=.,




( iv) Highlighting Syntax and validation  :-
a) PyFlakes
IMPORTANT: Your vim must have Python 2.5, at least. Enter ":python import sys; print sys.version" to see which version you have.
 The newest version of Vim, 7.3, comes bundled with Python 2.7 support.  pyflakes-vim highlights common Python errors like
misspelling a variable name on the fly. It also warns about unused imports, redefined functions, etc.

install details
1. Make sure your vimrc has "filetype plugin indent on"
so that pyflake-vim's ftplugin files are loaded automatically when you open a Python buffer.
2. Drop extracted files in ~/.vim/ftplugin/python.
3. That's it!

Download page link here  http://www.vim.org/scripts/script.php?script_id=2441
Direct unzip this pyflakes into ~/.vim
Config setting of .vimrc is below
let g:pyflakes_use_quickfix = 0

 b) Pep8        
install inside ~/.vim/ftplugin/python
Download page http://www.vim.org/scripts/script.php?script_id=2914
now switch to ~/.vim/ftlugin/python          
$cd ~/.vim/ftplugin/python          
$ wget -O pep8.vim http://www.vim.org/scripts/download_script.php?src_id=14366

Configuration for .vimrc  put below in your config file .vimrc

let g:pep8_map='<leader>8' set expandtab
set textwidth=79
set tabstop=8
set softtabstop=4
set shiftwidth=4
set autoindent    



v)Code Navigation :-
Catgs  this my best plugin for vim it can use for all programming language's and with this taglist is best for source code navigation .
Install a direct package only for Ubuntu else do manually is below
# apt-get install exuberant-ctags

first download ctags from here http://ctags.sourceforge.net/ .Ctags help you in while browsing a source code file:

Navigating to the function definition by specifying the function name.
Navigating to the function definition from ‘function call’.
Returning back again to function call from the definition.
Viewing the prototype/signature of functions or variables.
Viewing the number of functions in a file, etc., now extract ctags.tar.gz

then do this for configure and install ./configure && sudo make install

after that Config you .vimrc as below
let Tlist_Ctags_Cmd='/usr/local/bin/ctags'
set tags=./tags;/
map <C-\> :tab split<CR>:exec("tag ".expand("<cword>"))<CR>
map <A-]> :vsp <CR>:exec("tag ".expand("<cword>"))<CR>


Best use of this plugin with following command is below :- C-] - go to definition
C-T - Jump back from the definition.
C-W C-] - Open the definition in a horizontal split
C-\ - Open the definition in a new tab A-] - Open the definition in a vertical split

After the tags are generated. You can use the following keys to tag into and tag out of functions:

Ctrl-Left_MouseClick - Go to definition Ctrl-Right_MouseClick - Jump back from definition


Taglist :- with ctag for browsing source code better ,switch to ~/.vim/plugin
$ mkdir ~/.vim/plugin
$cd ~/.vim/plugin
$ wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701

direct download link is here http://www.vim.org/scripts/script.php?script_id=273

Config for .vimrc like below  " Taglist variables

" Display function name in status bar:
let g:ctags_statusline=1
" Automatically start script
let generate_tags=1
" Displays taglist results in a vertical window:
let Tlist_Use_Horiz_Window=0
" Shorter commands to toggle Taglist display
nnoremap TT :TlistToggle<CR>
map <F4> :TlistToggle<CR>
" Various Taglist diplay config:
let Tlist_Use_Right_Window = 1
let Tlist_Compact_Format = 1
let Tlist_Exit_OnlyWindow = 1
let Tlist_GainFocus_On_ToggleOpen = 1
let Tlist_File_Fold_Auto_Close = 1




Tasklist with Ctags for better manipulation


switch ~/.vim/plugin
download there this direct http://www.vim.org/scripts/script.php?script_id=2607
$ cd ~/.vim/plugin
$wget -O tasklist.vim http://www.vim.org/scripts/download_script.php?src_id=10388

Config .vimrc
map T :TaskList<CR>
map P :TlistToggle<CR>
use tasklist with T and P to use this



vi) Code Completion :-    
A) Python Omni  completion  place this in ~/.vim/autoload download direct link here
$ cd ~/.vim/autolaod http://www.vim.org/scripts/script.php?script_id=1542
$ wget -O pythoncomplete.vim http://www.vim.org/scripts/download_script.php?src_id=10872  

add your config to your .vimrc is below
filetype plugin on
set ofu=syntaxcomplete#Complete


B) Pydict for python code completion
download zip direct to ~/.vim/ftplugin  and extract  download page link here
http://www.vim.org/scripts/script.php?script_id=850

config your .vimrc with below stuff
let g:pydiction_location = 'C:/vim/vimfiles/ftplugin/pydiction/complete-dict'


C) Supertab for pydict easy completion :-
create a ~/.vim/after/ftplugin
download to it direct from this page http://www.vim.org/scripts/script.php?script_id=1643

Install this now using :- Open the file in vim ($ vim supertab.vba).

 Source the file (:so %)
use to press tab to complete the code

add in .vimrc
let g:SuperTabDefaultCompletionType = "context"




vii) Basic editing and debugging(pep8 ,pyflake is used above) ,code snippets:- 

A) Code snippets(snippetEMU) :-
The same thing works for “for” loops, classes and many other constructs. The bundles .vba provides access to such constructs across a wide range of languages, e.g. Python, Ruby, HTML, PHP, even Djano-specific constructs.

create a ~/.vim/after/ftplugin
download page link is here http://www.vim.org/scripts/script.php?script_id=1318

now install it above  open with vim snippy_plugin.vba then source the file (:so%)

B) snipMate:-it aims to be an unobtrusive, concise vim script that implements some of TextMate's snippets features in Vim. A snippet is a piece of often-typed text that you can insert into your document using a trigger word followed by a <tab>.   For instance, in a C file using the default installation of snipMate.vim, if you type "for<tab>" in insert mode, it will expand a typical for loop in C:  for (i = 0; i < count; i++) {  }

just download zip and extract in ~/.vim
Download page is here   http://www.vim.org/scripts/script.php?script_id=2540

C) Code folding
put this in ~/.vim/ftplugin
download from this page http://www.vim.org/scripts/script.php?script_id=1494

D)Block navigation :-This script can be useful when editing Python scripts.
It provides the following menus:
- Select a block of lines with the same indentation
- Select a function
- Select a class
- Go to previous/next class/function
- Go to the beginning/end of a block
- Comment/uncomment the selection
- Jump to the last/next line with the same indent

install details
Copy to the $VIMFILES/ftplugin directory download page is here

http://www.vim.org/scripts/script.php?script_id=30

E)Minibuffexpl.vim:-  download  here http://www.vim.org/scripts/script.php?script_id=159
paste in ~/.vim/plugin

config for .vimrc is below
let g:miniBufExplMapWindowNavVim = 1
let g:miniBufExplMapWindowNavArrows = 1
let g:miniBufExplMapCTabSwitchBufs = 1
let g:miniBufExplModSelTarget = 1




viii)Directory   Navigation:-Nerdtree download page  http://www.vim.org/scripts/script.php?script_id=1658

      install details
      Unzip the archive into your ~/.vim directory.
     That should put NERD_tree.vim in ~/.vim/plugin and NERD_tree.txt in ~/.vim/doc.


ix)Git integration with vim :- download page is here http://www.vim.org/scripts/script.php?     script_id=2975
    just extract zip in ~/.vim    

x) For automatic closing of quotes, parenthesis, brackets:-   delimitMate
    Download page is here http://www.vim.org/scripts/script.php?script_id=2754
 
   download this in ~/.vim/after/ftplugin , Now install using below

   Open the file with Vim and run:
   :UseVimball

   or use this command to install it in your bundle dir if you use pathogen:
   :UseVimball ~/.vim/bundle/delimitMate /              



Note:- Stay Tuned more parts of this Post coming soon for Ruby,C++,C,Closure,Prolog ,Haskell,Java,PHP,Perl,Shell, JavaScript ,HTML,CSS Coming soon (within few Days) .
Update of this post will done soon for Usage of plugin and more config Details .


THANKS FOR VISITING


LHS AS A SOURCE OF INFORMATION – AND A SOURCE OF INSPIRATION – I HOPE YOU’LL CHOOSE TO ACT RIGHT NOW.ENJOY KEEP LEARNING.




Monday, January 2, 2012

Large Scale & Big Data analysis with Hadoop cluster (Using Ubuntu 11.10 server)

Welcome back guys, as I promised to a setup of Hadoop cluster on Ubuntu Server(64 bit)
last time I posted a blog about OpenStack and Devstack.

While everybody celebrating New Year, I was working on setting up a Hadoop cluster and processing unstructured data.


Purpose
The purpose of this document is to help you get a single-node Hadoop installation up and running very quickly so that you can get a flavor of the Hadoop Distributed File System (see HDFS Architecture) and the Map/Reduce framework; that is, perform simple operations on HDFS and run example jobs.


Our Pre-requisites for setup:-
1) We Must need a Ubuntu Server (64 bit) or Debian (64bit ) {I am using same this tutorial}
2)  You must have there installed (in the main root)
     $ sudo apt-get install ssh (openssh )  
     $ sudo apt-get install rsync
  
3) JavaTM 1.6.x, preferably from Sun, must be installed.

    Installing Java
 
    $ sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
    $ sudo apt-get update
    $ sudo apt-get install sun-java6-jdk sun-java6-plugin  ( while after Downloading  during installation plz press tab to accept JAVA terms and condition while a tab for conditions open )
A check Sun java is there :)


root@ruhil:~# sudo apt-get install sun-java6-jdk sun-java6-plugin
Reading package lists... Done
Building dependency tree   
Reading state information... Done
sun-java6-plugin is already the newest version.
sun-java6-jdk is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.
root@ruhil:~# java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
root@ruhil:~#


Add a Dedicated User

$ sudo addgroup hadoop
$ sudo adduser --ingroup hadoop hduser


Configure and check is ssh working for local-host
(Please press the enter , you need not to specify the name  for File for  and Public key )
root@ruhil:~# su - hduser
hduser@ruhil:~$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa):
Created directory '/home/hduser/.ssh'.
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.
The key fingerprint is:
9b:82:ea:58:b4:e0:35:d7:ff:19:66:a6:ef:ae:0e:d2 hduser@ruhil
The key's randomart image is:
[...snipp...]
hduser@ruhil:~$

After doing this step carefully

hduser@ruhil:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_key

hduse@ruhil:~$ ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is b8:be:26:41:44:7d:9b:82:02:fd:13:61:3c:ac:d4:0a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Linux ruhil 3.0.0-14-server #23-Ubuntu SMP Mon Nov 21 20:49:05 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
[...snipp...]
hduser@ruhil:~$



IN Ubuntu 11.10
open /etc/sysctl.conf in the editor of your choice and add the following lines to the end of the file:

#disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

You have to reboot your machine in order to make the changes take effect.

You can check whether IPv6 is enabled on your machine with the following command:

$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6

A return value of 0 means IPv6 is enabled, a value of 1 means disabled (that’s what we want).



Installation of Hadoop (perform this action in your main root like root@ruhil)
{Note Downloading 0.20 version is stable other not stable mainly 0.23,Would like go with 0.20  :)}
root@ruhil:~# mkdir -p /usr/local
root@ruhil:~# cd /usr/local
root@ruhil:~# wget http://apache.mirrorcatalogs.com/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz -O hadoop-0.20.2.tar.gz
root@ruhil:~# sudo tar xzf hadoop-0.20.2.tar.gz
root@ruhil:~# mv hadoop-0.20.2 hadoop
root@ruhil:~$ sudo chown -R hduser:hadoop hadoop



Create.Bashrc or If have already pasted below for Hadoop(Note: -you need paste in root and hduser ,if you like you can paste for all, My Paste of .bashrc is http://paste.ubuntu.com/791667/):-

# Set Hadoop-related environment variables
export HADOOP_HOME=/usr/local/hadoop

# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)
export JAVA_HOME=/usr/lib/jvm/java-6-sun

# Some convenient aliases and functions for running Hadoop-related commands
unalias fs &> /dev/null
alias fs="hadoop fs"
unalias hls &> /dev/null
alias hls="fs -ls"

# If you have LZO compression enabled in your Hadoop cluster and
# compress job outputs with LZOP (not covered in this tutorial):
# Conveniently inspect an LZOP compressed file from the command
# line; run via:
#
# $ lzohead /hdfs/path/to/lzop/compressed/file.lzo
#
# Requires installed 'lzop' command.
#
lzohead () {
    hadoop fs -cat $1 | lzop -dc | head -1000 | less
}

# Add Hadoop bin/ directory to PATH
export PATH=$PATH:$HADOOP_HOME/bin




Configuration(Note all the configuration setting you need to be made in hduser):-

The following picture gives an overview of the most important HDFS components.HDFS Architecture (source: http://hadoop.apache.org/core/docs/current/hdfs_design.html)

Our goal in this tutorial is a single-node setup of Hadoop. More information of what we do in this section is available on the Hadoop Wiki.
hadoop-env.sh

The only required environment variable we have to configure for Hadoop in this tutorial is JAVA_HOME. Open /conf/hadoop-env.sh in the editor of your choice (if you used the installation path in this tutorial, the full path is /usr/local/hadoop/conf/hadoop-env.sh) and set the JAVA_HOME environment variable to the Sun JDK/JRE 6 directory.

Change

# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun

to

# The java implementation to use.  Required.
export JAVA_HOME=/usr/lib/jvm/java-6-sun



Now we create the directory and set the required ownerships and permissions:
$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
$ sudo chmod 755 /app/hadoop/tmp
{Set Your chmod according to your settings  }


Add the following snippets between the <configuration> ... </configuration> tags in the respective configuration XML file.
Note for all given below we need perform all this  below config files


In file conf/core-site.xml:(cd /usr/local/hadoop there all this config )
<!-- In: conf/core-site.xml -->
<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>

In file conf/mapred-site.xml:

<!-- In: conf/mapred-site.xml -->
<property>
  <name>mapred.job.tracker</name>
  <value>localhost:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>

In file conf/hdfs-site.xml:

<!-- In: conf/hdfs-site.xml -->
<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>



hduser@ruhil:~$ /usr/local/hadoop/bin/hadoop namenode -format
The output will look like this:
hduser@ruhil:/usr/local/hadoop$ bin/hadoop namenode -format
1/01/12 1:30:41 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ruhil/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'ruhil' on Sun Jan 1 01:30:41 UTC 2012
************************************************************/
1/01/12 1:30:41 INFO namenode.FSNamesystem: fsOwner=hduser,hadoop
1/01/12 1:30:41  INFO namenode.FSNamesystem: supergroup=supergroup
1/01/12 1:30:41  INFO namenode.FSNamesystem: isPermissionEnabled=true
1/01/12 1:30:41  INFO common.Storage: Image file of size 96 saved in 0 seconds.
1/01/12 1:30:41  INFO common.Storage: Storage directory .../hadoop-hduser/dfs/name has been successfully formatted.
1/01/12 1:30:41 5/08 16:59:57 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ruhil/127.0.1.1
************************************************************/
hduser@ruhil:/usr/local/hadoop$


Starting your single-node cluster
hduser@ruhil:/usr/local/hadoop/bin/start-all.sh
hduser@ruhil://usr/local/hadoop$ jps
all this shown below in bash too





Stop Hadoop using below command:-
hduser@ruhil:~$ /usr/local/hadoop/bin/stop-all.sh
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: stopping datanode
localhost: stopping secondarynamenode
hduser@ruhil:~$


Now Run a Map-reduce job:-
Just watch bash carefully







NOW Finally Look into your Browser Yeah:-)

Hadoop Web Interfaces

Hadoop comes with several web interfaces which are by default (see conf/hadoop-default.xml) available at these locations:















I hope you enjoyed it :), process your data with ease and super speed :)
Looking for any kind Help (on Hadoop IRC  with #cloudgeek)
Feel free to mail me vikasruhil06@gmail.com 

FOLLOW US OF TWITTER 

THANKS FOR VISITING


LHS AS A SOURCE OF INFORMATION – AND A SOURCE OF INSPIRATION – I HOPE YOU’LL CHOOSE TO ACT RIGHT NOW. ENJOY KEEP LEARNING.






















                        



             

Friday, December 16, 2011

E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable),How to fix (Solved)

As a administer, newbie UNIX/Linux user ,or developer/programmer  , you face a problem like this in your shell/bash  in Ubuntu/Debian when any package is broken or due to any reason  your get lock /var/lib/dpkg/lock. or like below you face problem


root@pythongeek:~# apt-get update
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?

you can fix this problem in few steps without rebooting !!


1) type this in bash

    sudo dpkg --configure -a

2) type in bash

    sudo killall apt-get apt aptitude adept synaptic

your output like this

root@pythongeek:~# sudo killall apt-get apt aptitude adept synaptic
apt-get: no process found
apt: no process found
aptitude: no process found
adept: no process found
synaptic: no process foun

Note:if you face problem during installation of any software/package

don't forget accept EULA ,or accepting licensee of user agreement while you have pop-up during installation
 
3) this is final if you failed last to attempt to unlock
Close all running packages, and open a Konsole window.

type in bash

sudo rm /var/lib/dpkg/lock

then again type this bash

sudo dpkg --configure -a

for reinstallation type like this in bash

sudo apt-get  install -f  package  name here


FOLLOW US OF TWITTER 

THANKS FOR VISITING


LHS as a source of information – and a source of inspiration – I hope you’ll choose to act right now.enjoy keep learning.


Sunday, December 11, 2011

Installing openstack on Ubuntu using Devstack with ease in 15 min only (Cloud computing part 2)

Devstack script is a useful tools and good tutorial for us.

It help me know how to install openstack(nova, glance, keystone and so on) from git. And teach us how to config them and make them work fine together. Now using devstack script is very easy if you just want to set up an openstack environment for learning.

Remember you need not install mysql and anything else in advance it will automatically installed from script   

Make sure your system is 11.10 Ubuntu server
you can test it in your really machine or in your vmware or virtualbox desktop machine.



Make sure you open your virtualization switch in your BIOS setting.
you need have to two interfaces, ie eth0 and eth1
--------------------------------------------------------------------------------------
$ sudo apt-get install bridge-utils     # install birdge
$ vi /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
address 10.11.3.62
netmask 255.255.255.0
network 10.11.3.0
broadcast 10.11.3.255
gateway 10.11.3.1

auto br100
iface br100 inet static
bridge_ports eth2
bridge_stp off
bridge_maxwait 0
bridge_fd 0
address 10.200.200.1
netmask 255.255.255.0



in this file, eth0 is your public network, you need it to connect to internet. eth2(maybe in your system is eth1) is another network interface used as a bridge(br100), 10.200.200.1 is your private ip, it is not need connect to any other network in our test.just choose a private network like 10.200.200.0/24 and remember.


$ sudo apt-get install git
$ git clone git://github.com/cloudbuilders/devstack
$ cd devstack
$ vi localrc
add these info into localrc


PUBLIC_INTERFACE=eth0
FIXED_RANGE=10.200.200.0/24
FIXED_NETWORK_SIZE=256
FLOATING_RANGE=10.11.3.0/24
NET_MAN=FlatDHCPManager
FLAT_NETWORK_BRIDGE=br100
VIRT_DRIVER=libvirt
LIBVIRT_TYPE=kvm
SCHEDULER=nova.scheduler.simple.SimpleScheduler



$ ./stack

it will ask for some password, remeber don't use special characters like $*_.! and so on. you can simply set password like 123456
then it will begin install , how much it take depends on your network bandwidth. If it failed since of network or some reason. look at error log carefully, delete /opt/stack/devstack and run ./stack again.




At last, you will see successful information, and then you can type your IP in your browser. Enjoy your open-stack time and it is a good thing you need learning more and more. finally you are here .

Looking for Help/ Technical support  !

Feel free to mail me vikasruhil06@gmail.com

FOLLOW US OF TWITTER 

THANKS FOR VISITING


LHS as a source of information – and a source of inspiration – I hope you’ll choose to act right now.enjoy keep learning.






Creating a Private Cloud using Openstack in Ubuntu Maually (Cloud Computing Part 1)

This article dedicated to developer ,project leaders , Cloud architects . I am gonna explain how to setup a cloud using command line .This useful for newbies to cloud and UNIX both !

I am using a Ubuntu 11.10 (64 bit server ), so lets play with shell !!

for setting up a prefect cloud you need minimum two network NIC on your hardware , a lot of public Ip . :)

But i am setting it on one network  it on one NIC hardware , with One IP (students and tester can with way those have less resources ) :)

We are setting up it with open-stack .
Official website of this document is below:

ubuntu 11.10 environment deployment, select the network mode FLATDHCP

public interface: eth0 used to connect to the user
private interface: eth1 do bridge br100, used, and other nodes, keystone, glance, volume and other connections

Minimum installed Ubuntu

Remember to open-ssh installed, where we have established a common user open-stack

Giving it the NOPASSWD sudo privileges, so easy to operate behind

$ Sudo apt-get update          //  (apt update about the tree)
$ Sudo apt-get install bridge-utils     //   (  installed bridge components )

Configure the network
$ vi / etc / network / interfaces      //(gksudo also can be used instead of vi for GUI Fans )

auto eth0
iface eth0 inet static
address 192.168.200.21
netmask 255.255.255.0
network 192.168.200.0
broadcast 192.168.200.255
gateway 192.168.200.10

auto br100
iface br100 inet static
bridge_ports eth1
bridge_stp off
bridge_maxwait 0
bridge_fd 0
address 10.200.200.2
netmask 255.255.255.0

$ Sudo / etc / init.d / networking restart

Initial preparatory work to do, the next step is to install on nova, glance and other components

$ Sudo apt-get install-y rabbitmq-server  // (install the MQ message components )
$ Sudo apt-get install-y python-greenlet python-mysqldb     // (install Python dependencies )

Next, install the various nova components and dependencies


$ Sudo apt-get install nova-volume nova-vncproxy nova-api nova-ajax-console-proxy
$ Sudo apt-get install nova-doc nova-scheduler nova-objectstore
$ Sudo apt-get install nova-network nova-compute
$ Sudo apt-get install glance

Installation euca2ools and unzip
$ Sudo apt-get install-y euca2ools unzip

Next we install the database, I chose MYSQL, PostgreSQL actually personally feel better
$ Sudo su - to root user to change
# MYSQL_PASS = nova nova set mysql database password and the password
# NOVA_PASS = notnova here nova and notnova modified according to their definitions
# Cat <<MYSQL_PRESEED | debconf-set-selections
> Mysql-server-5.1 mysql-server/root_password password $ MYSQL_PASS
> Mysql-server-5.1 mysql-server/root_password_again password $ MYSQL_PASS
> Mysql-server-5.1 mysql-server/start_on_boot boolean true
> MYSQL_PRESEED
# Apt-get install-y mysql-server
# Exit exit root environment

$ Sudo sed-i 's/127.0.0.1/0.0.0.0/g' / etc / mysql / my.cnf modify my.cnf configuration file
$ Sudo service mysql restart

$ MYSQL_PASS = nova in the general user environment variable to the password once again set about
$ NOVA_PASS = notnova
$ Sudo mysql-uroot-p $ MYSQL_PASS-e 'CREATE DATABASE nova;'

// (to create a name for the ova of the database, I recommend that new users nova's name, if here for another name, then the configuration file in the nova which also need to change )
$ Sudo mysql-uroot-p $ MYSQL_PASS-e "GRANT ALL PRIVILEGES ON *.* TO
'Nova'@'%' WITH GRANT OPTION; "
$ Sudo mysql-uroot-p $ MYSQL_PASS-e "SET PASSWORD FOR 'nova'@'%' =
PASSWORD ('$ NOVA_PASS'); "

This point. nova, glance of the installation completed, next is the configuration

nova configuration
$ Sudo vi / etc / nova / nova.conf
- Dhcpbridge_flagfile = / etc / nova / nova.conf
- Dhcpbridge = / usr / bin / nova-dhcpbridge
- Logdir = / var / log / nova
- State_path = / data / openstack / nova here / data / openstack / nova is a new volume and directory, make sure you have this, and the user should belong to nova
- Instances_path = / data / openstack / nova / instances to modify the default storage of instances where
- Lock_path = / var / lock / nova
- Force_dhcp_release = True
- Use_deprecated_auth
- Iscsi_helper = tgtadm
- Verbose
- Scheduler_driver = nova.scheduler.simple.SimpleScheduler
- Network_manager = nova.network.manager.FlatDHCPManager
- My_ip = 10.200.200.2 This is my ip address within the network
- Public_interface = eth0
# - Vlan_interface = eth0
- Sql_connection = mysql: / / nova: notnova @ localhost / nova
- Libvirt_type = kvm
# - Osapi_extensions_path = / opt / nova / bin / openstackx / extensions
# - Vncproxy_url = http://10.200.200.2:6080
# - Vncproxy_wwwroot = / data / stack / noVNC /
- Api_paste_config = / etc / nova / api-paste.ini
- Image_service = nova.image.glance.GlanceImageService
- Ec2_dmz_host = 192.168.200.21
- Ec2_url = http://192.168.200.21:8773/services/Cloud
- Rabbit_host = localhost
- Glance_api_servers = 10.200.200.2:9292
- Flat_network_bridge = br100
- Flat_interface = eth1
- Flat_network_dhcp_start = 10.200.200.51 specified instances allocated from the beginning from the 51, but looks like this option does not work
- Fixed_range = 10.200.200.0/24 This option specifies the instances of the network segment
- Flat_injected = False
- Multi_host = 1 using multi_host
- Libvirt_use_virtio_for_bridges instances using virtio network card model do
# - Start_guests_on_host_boot = true
# - Resume_guests_state_on_host_boot = true
- Use_ipv6 = false

$ Sudo vi / etc / glance / glance-api.conf
// (In this file without using keystone of the present case, according to your needs, modify filesystem_store_datadir parameter to specify the directory you need to store images, Dangran are the main users have Gaicheng glance )
$ Sudo vi / etc / glance / glance-registry.conf
// (This file can be selected to modify sql_connection parameter to specify your database. Of course you can not change.
If you want to modify, use mysql which database to ensure that the mysql which established a corresponding database )

sql_connection = mysql: / / nova: notnova @ localhost / glance this is my configuration, I created a glance in the mysql database

$ Sudo chown-R root: nova / etc / nova to change / etc / nova's owner
$ Sudo chmod 640 / etc / nova / nova.conf

Restart all services
$ Sudo restart libvirt-bin
$ Sudo restart nova-network
$ Sudo restart nova-compute
$ Sudo restart nova-api
$ Sudo restart nova-objectstore
$ Sudo restart nova-scheduler
$ Sudo restart glance-registry
$ Sudo restart glance-api
Note: We do not have from the nova-volume, because although we installed the volume, but the volume needs to use a separate vg, we have not configure the volume, so get up.

There may be network services and can not compute it, do not worry about being first

Next, we do configure the operating environment of the nova
$ Sudo nova-manage db sync
nova-manage user admin <user_name> where we can create a user, such as
$ Sudo nova-manage user admin test, create a successful return on the screen like:
export EC2_ACCESS_KEY = d6aa7747-4324-4abc-9604-4f7d6a2f8f3f
export EC2_SECRET_KEY = 2b204b75-da2d-47b8-ba7a-611d71f0ecbf

nova-manage project create <project_name> <user_name> create a project, we built that are just users, such as:
$ Sudo nova-manage project create test-proj test
nova-manage network create - help create an instance of the network, such as:
$ Sudo nova-manage network create - label = test-net - fixed_range_v4 = 10.200.200.0/24 - num_network = 1 - network_size = 256

Services have failed to start again
$ Sudo start nova-network
$ Sudo start nova-compute
$ Sudo start nova-scheduler
In addition, since each service, the best look at the log, such as sudo tail-f / var / log / nova / nova-network to determine there is no error, you can also use the ps aux | grep [n] ova-network to confirm the service is not open. If the starting service fails, you confirm that a good reason to modify a good future, need to use sudo start to play instead of sudo restart

Well. This computing environment, we deployed the. We can look at the state command
$ Sudo nova-manage service list
$ Sudo nova-manage network list
Etc.

Next, create a certificate, to facilitate the tool we use euca
$ Cd
$ Mkdir creds
$ Sudo nova-manage project zipfile test-proj test creds / novacreds.zip
$ Unzip creds / novacreds.zip-d creds /
$ Source creds / novarc

OK, done, we can use the tool to look at
$ Euca-describe-availability-zones verbose
VAILABILITYZONE nova available
AVAILABILITYZONE | - Nova-test
AVAILABILITYZONE | | - Nova-network enabled :-) 2011-10-17 04:45:44
AVAILABILITYZONE | | - Nova-compute enabled :-) 2011-10-17 04:45:45
AVAILABILITYZONE | | - Nova-scheduler enabled :-) 2011-10-17 04:45:46

So far, successfully enabled services. When you find that service is not working, use ps aux | grep nova check services are not open, and the need for detailed observations / var / log / nova / directory log files for each service, in order to obtain further information.


Then we can use kvm to create a mirror image

$ Sudo apt-get install kvm-pxe installation about this, otherwise there will be time to run kvm warning
$ Kvm-img create-f raw server.img 5G
$ Sudo kvm-m 1024-cdrom rhel5.iso-drive file = server.img, if = virtio, index = 0-boot d-net nic-net user-nographic-vnc: 0

Here we use rhel5(Redhat) the iso, after running this command, you can use vnc to connect the machine to connect to your server: ssvncviewer 192.168.200.21: 0
Open vnc you can see the installation interface After the installation, the following paragraph written rhel mirror / etc / rc.local the beginning of the
depmod-a
modprobe acpiphp

# Simple attempt to get the user ssh key using the meta-data service
mkdir-p / root / .ssh
echo>> / root / .ssh / authorized_keys
curl-m 10-s http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key | grep 'ssh-rsa'>> / root / .ssh / authorized_keys
echo "AUTHORIZED_KEYS:"
echo "************************"
cat / root / .ssh / authorized_keys
echo "************************"
Save and exit, so that image on the well

Upload images using glance
$ Glance - verbose add name = "rhel5" disk_format = raw is_public = true <server.img
You should also observe the / var / log / glance / registry and the log api
$ Glance index to see a list of mirrors

Start your instance
$ Euca-describe-images
You can now view the image, output similar to the
IMAGE ami-00000003 server.img
Remember this image here ami-000000003 No.
$ Euca-run-instances-t m1.tiny ami-00000003 start an instance of ami-00000003 image
-T specifies the type of instance, provides the type of cpu, memory, disk size, etc..
Watch / var / log / nova / nova-api.log nova-scheduler.log, nova-compute, nova-network.log of output, but you can also use vnc to connect serverip: 0 look at the console instance
With the command $ euca-describe-instances to see your current instance of the first instance will be relatively slow start because of the need to copy the image from a glance under the instance directory to the nova

Conclusion As the nova is currently growing fast, diablo release version of the function of some of the requirements to be completed daily. But the development version of the nova can be better combined with keystone, novaclient, dashboard and some other projects. Makes openstack more robust. Friends who are interested, you can use in a production environment repo's installation, test development versions of the test environment. Since I use in a production environment install git development version. Therefore, a more complete integration of follow-up, I will develop versions of the form. Of course, using the development version, then there will be more trouble, but also have more fun and hands-on practice, to further understand the mechanism of its working principle.

Looking for help /technical support  !

Feel free to mail me vikasruhil06@gmail.com



FOLLOW US OF TWITTER 

THANKS FOR VISITING


LHS as a source of information – and a source of inspiration – I hope you’ll choose to act right now.enjoy keep learning.