This is great at first but after a while log files become enormous and eat up lots of disk space. The problem multiplies when you start getting into a clustered environment (more than 1 server). You now have to read & maintain logs on 2+ machines which will most likely lead to neglecting your error fixing duties. Fortunately MongoDB recently hit the scene.

MongoDB

MongoDB is a scalable, high-performance, open source, document-oriented database. It’s ability to accept huge amounts of data very quickly make it perfect for application logging. It has built in support for auto-sharding & replication so when your application takes off mongodb will scale with you. It uses a JSON based storage system meaning you can give the database an object (from any programming language with support) and when you pull it back out of the database it will be the exact same as when you put it in. No conversion, no hassle, no bullshit.

Capped Collections

Mongo uses things called Collections which are essentially tables in mysql. The main difference is these collections are schemaless. I can have 10 fields in one document (row) and 500 in another and they will play nice inside the collection. This allows a huge amount of flexibility to the developer.

These collections also have the ability to be “Capped”. This means I can set a max document count of 5000 and only the most recent 5000 documents will be kept. The older documents are deleted as new ones come in. This is awesome for logging because you will never have to rotate your logs in fear that they will become too large and occupy all of your file system space.

Use this command to cap a collection (in a mongo shell):

db.createCollection("my_collection", {capped:true, size:5000})

More Information on Capped Collections

Use in CodeIgniter

By default CodeIgniter writes to log files in /system/logs/ . I have created a class that overwrites that default functionality and sends all logs to a mongo db collection. Adding this library to all of your servers will allow all of those servers to write to the same mongo collection. I have also added some more data to the logs that is helpful in a clustered environment such as the server name, server ip, request uri, and a few more. Just place the MY_Log.php in your application/libraries directory and codeigniter should handle the rest.

You can view and fork the class on Github:
http://github.com/trs21219/codeigniter-mongo-logs/

Feel free to modify data inserted to fit your specific needs. If you find any bugs or have suggestions you can comment below or raise an issue here.

Probably the biggest challenge for a newly popular site these days is scalability. The most common point of failure in applications seems to always be the database. These days our applications are totally reliant on some sort of relational database to store the information for our website’s content. Each and every page load brings many queries that read or insert information into this database all while rendering the web page in a timely manner. This isn’t so much a problem for your Mom & Pop business, but when your website starts to get some real traffic pointed towards it,  your underlying architecture that you thought was impervious to anything will crumble before your eyes. The solution? Stop making so many queries to your database.

Caching

Caching is nothing new. Servers have been caching content for many years quite successfully. There are many different ways you can cache data in your application but right now we will just be talking about using Memcached. Memcached is a high-performance, distributed memory object caching system. So what does this mean? Basically it can store any kind of value that you would normally store in your database and retrieve it much much faster than your database can.

Memcached Flow

When you implement memcached you use it along side of your database. Neither takes the place of each other, memcached just takes some of the load off of your database server after your database server has previously done the grunt work. It works on a key pair system. Meaning it has no tables, rows, columns etc. You give memcached a name and pass it data and it stores that data just as you passed to it. You can pass it an object, array, binary, etc and it will output the same way when you go to retrieve it.

So how does this all work with the database? First, you add code that checks to see if the memcached object (blogpost-45 for our example) exists before it checks the database for the data. If it does you return that data and never touch the DB.  If the memcached object does not exist  you retrieve data from your database, take the array that is passed back and store it in memcached with the key of “blogpost-45″. So the next time you try to get the data for blogpost-45 it will return from memcached and not your database. This is where the load savings occurs.

So what happens when I want to change the data? When you change the data in your database you should delete the key in memcached. You shouldn’t re-add the data after you delete it, let the select query do that to avoid complications.

CodeIgniter + Memcached

I have created a CodeIgniter Library that can help you get a jump start on your integration with memcached, its available on my github page and will be updated when I get time.

http://github.com/trs21219/memcached-library

Security

Through the development cycle of a web application, you are going to write hundreds if not thousands of queries for each specific thing you want your application to do. That means for each query you write you will have to wrap it with functions that make it safe so that SQL injection and other malicious things cannot take place. Now say if you write 1000 queries through he course of your development and you forget to properly escape them 1% of the time (very very conservative). You will have 10 possible places for your malicious users to bring down your system.

If you had used Active Record you wouldn’t have had that happen. Active Record doesn’t forget to escape and filter those queries. You wouldn’t have a client calling you at 3am because their pride and joy is displaying porn ads or even worse that their customer’s personal information has been leaked to hackers.

Performance

One of his concerns was the performance impact that the Active Record class imposed. Coming from someone who uses the AR class every day in my 9-5, I have never noticed a significant performance impact on any of the queries I write. I dont know the exact benchmarks of it but I would think the most it adds to any query would by around .0001 seconds. I think the small impact it has on your application is well worth the peace of mind that all of your queries are secure from the few malicious users you are bound to cross paths with.

Complex Queries

The Active Record class does a pretty good job at 99% of the queries you are most likely to run. It powers through insert/update/select/delete queries with no problem at all. However it is not perfect. If your application demands a few complex queries you can very easily switch back to straight SQL with $this->db->query(); . This way you get the best of both worlds. You can be secure in your cookie cutter queries that you write all the time but when you have to do something that is a little more complicated you have the ability to switch very easily.

In my opinion there isn’t a real reason not to use CodeIgniter’s Active Record. The benefits it provides far outweigh the downsides. Unless of course your someone who never makes a mistake in programming, in that case forget everything I have told you.

Ever since I started using CodeIgniter about a year and a half ago, I feel that my coding style has drastically improved as well as the speed, security, and reliability of my applications. This is mostly due to CodeIgniter’s MVC approach to the structure of the application. It separates your code into three stages:

• Model – Used to interact with your database or data source and return the data back to your application
• View – Holds all of the display logic such as html, css and javascript
• Controller – Sort of the middleman, used to coordinate all the actions from Models and Libraries into a View to display to the user’s browser

CRUD

CRUD (Create, read, update and delete) is a way of structuring your code that interfaces with your database. Those four actions handle 99% of the actions you will do with the data in your database so why should we duplicate functionality every time we need to do something? For my examples I will be using the Active Record Class in CodeIgniter but it would be just as easy for you to use simple SQL statements in your PHP code.

Ok that’s enough of the vocabulary lesson, lets get into some code.

Start of the Model

All we are doing here is declaring the model as “Blog_model” and we will save this code in blog_model.php in the models folder in Codeigniter.

Create

Creating data is pretty simple, especially in codeigniter. You should only need a single function to create data for each table in your database. This is how i structure my insert functions.

Now what the code above is actually doing is this. When i call $this->whatever_model->insert_blog_post($data); , i pass a $data variable which is actually just an array with the index names matching to the database field names. If you are familiar with CodeIgniter’s Active Record Class this approach is nothing new to you.

Read

Reading information from a database is nothing new. You send a select query to a database and it returns results, how could it get any simpler? Well what usually ends up happening over the development of an application is that the developer will create multiple functions that essentially do the same thing. One function will query the database based on the Unique ID # of an item. Another will query checking to see if a particular title in that blog post already exists. So how can these be combined? Well you can create a single function that actually interacts with the database. It is passed a few parameters that tell it what query to run and it does just that. This function is called by other functions that pass those specific parameters. Here is an example.

Update

Now updating an row (or many rows) in the database is just as easy as adding a row. The first parameter is your where statement which includes the conditions that the rows must have to be updates. The second parameter is the data that will be inserted into those selected rows.

Pretty simple right? Now all you have to do from anywhere in your application to update a row in blog_posts is $this->blog_model->update_blog_post(array(‘ID’ => 5), array(‘title’ => ‘Getting Started with CRUD’));

Delete

Just as you might suspect, deleting a post is just as simple as adding or updating a post.

Now just run the command $this->blog_model->delete_blog_post(array(‘ID’ => ’223′));

Thats it!

If you have any questions or comments be sure to leave them below!

I use Mozenda all the time. When scraping crappy static html pages it is very difficult to find consistency with markup. So thats where Mozenda comes in. I create an agent to scrape a collection of data in a specific order. It can select data between html tags, select data thats supposed to be in one featured place on the page, or it can skip over it if the data you are looking for is not there. It is truely a lifesaving service for anyone that updates mass amounts of data on a regular basis. You cant beat the price of $49/month either (just think how much you would have to pay someone to scrape all of that data for you). I have created a CodeIgniter Library (regular php class compatible) to interact with this api in a easy way. Now lets get to it!

Download

Documentation

First off you need to set your api key and set what format you would like the results to be returned to you in. If you don’t have an api key then

1. Login to your account at http://Login.Mozenda.com

2. Click the ‘Account’ link located in the top right corner of the web page.

3. Look for the ‘API Web Service Key’ section under the ‘Account Details’ tab, then click ‘Generate a New Key’. You will be required to provide this key in all requests to the API.

If you already have your API key then all you need to worry about is setting your output format. The two options are array (php array) and json.

Configuring Your Library

[code language='php']
$config_array = array('output_format' => 'json', 'api_key' => 'MY-SUPER-SECRET-KEY');
$this->mozenda_api->config($config_array);
[/code]

Collection.GetList

Returns a list of collections for an account

[code language='php']

$data = $this->mozenda_api->collection_get_list();

[/code]

Collection.GetViews

Gets a list of views for a particular collection

[code language='php']

$data = $this->mozenda_api->collection_get_views($collection_id);

[/code]

Collection.GetFields

Returns a list of fields that are in that collection with their details

[code language='php']

$data = $this->mozenda_api->collection_get_fields($collection_id);

[/code]

Collection.AddItem

Adds an item to a collection with the values specified.

[code language='php']

$items = array('Username' => 'John', 'Phone_Number' => '555-0123');

$data = $this->mozenda_api->collection_add_item($collection_id, $items);

[/code]

Collection.UpdateItem

Updates an item in the collection.

[code language='php']

$items = array('Username' => 'Peter', 'Phone_Number' => '555-9876');

$data = $this->mozenda_api->collection_update_item($collection_id, $item_id, $items);

[/code]

Collection.DeleteItem

Deletes an item from a collection.

[code language='php']

$data = $this->mozenda_api->collection_delete_item($collection_id, $item_id);

[/code]

Collection.Clear

Clears the contents of a collection but leaves the collection intact.

[code language='php']

$data = $this->mozenda_api->collection_clear($collection_id);

[/code]

Collection.Delete

Deletes the collection and all data within it.

[code language='php']

$data = $this->mozenda_api->collection_delete($collection_id);

[/code]

View.GetItems

Returns items from a view.

[code language='php']

$data = $this->mozenda_api->view_get_items($view_id);

[/code]

Agent.GetList

Returns a list of your agents with their ID, Name, Settings, Description, and other important information.

[code language='php']

$data = $this->mozenda_api->agent_get_list();

[/code]

Agent.GetJobs

Returns a list of your agent’s jobs with detailed information.

[code language='php']

$data = $this->mozenda_api->agent_get_jobs($agent_id);

[/code]

Agent.Run

Starts or resumes the Agent.

[code language='php']

$data = $this->mozenda_api->agent_run($agent_id);

[/code]

Agent.Delete

Deletes an agent and all associated schedules for that agent.

[code language='php']

$data = $this->mozenda_api->agent_delete($agent_id)

[/code]

Job.Get

Gets the details of a job by the Job ID.

[code language='php']

$data = $this->mozenda_api->job_get($job_id);

[/code]

Job.Cancel

Cancels a Job in the system. Note, a job must be in a Paused or Error State to cancel a job.

[code language='php']

$data = $this->mozenda_api->job_cancel($job_id);

[/code]

Job.Pause

Issues the ‘Pause’ command for a job currently running in the system.

[code language='php']

$data = $this->mozenda_api->job_pause($job_id);

[/code]

Job.Resume

Resumes a job that is in a Paused or Error state.

[code language='php']

$data = $this->mozenda_api->job_resume($job_id);

[/code]

I would really recommend taking a look at the official documentation on the mozenda website show here.

That’s It!

That pretty much wraps it all up. If you have any questions or find a bug please use the comments and ill do my best to fix it.

NOTE: The descriptions of what the commands do was copied from the mozenda documentation to make it easier for the end user to understand. I do not take any credit in writing the descriptions.

So I created this library mostly out of boredom. It includes functions to interact with all of the options & functionality that the current version (“EDGE”) provides. It was designed to be a CodeIgniter library but it can easily be ported over to php in just a few steps. It allows you to output the data in json, array, or raw (the format that the request was [json, xml]). So lets get to it!

Download

[download id="1"]

Documentation

Configuration

There are several options you can set in the configuration that will make the library behave differently.

• username – obviously your username for the envato marketplaces
• api_key – your api key for the marketplaces ( you should never set this in view of the public, its sort of like a password ).
• api_format – the format you want the library to request, this can be altered with the next option (output format) – options incude : json, array, xml (default is json)
• output_format – the format you want the library to return to you. options: array (default), json, raw (unaltered/formatted data straight from the feed) note: to get xml format back you need to set this to raw and set api_format to xml

This is how you can set the config information with a simple function and array.
[code language='php']
$this->envato_api->config(array('api_format' => 'json', 'output_format' => 'array', 'username' => 'joesmith', 'api_key' => 'MYSUPERSECRETKEY'));
[/code]

Place the configuration code above the functions listed below.

Public Functions

These are functions that do not require a username and api key to access. They are mostly things such as the latest files on the marketplaces, the most popular threads, etc.

Blog Posts

A list of blog posts for a particular site. Requires a site parameter.

[code language='php']
$data = $this->envato_api->public_blog_posts('sitename');
$this->evanto_api->clear();
[/code]

Active Threads

Threads with the most recent messages activity. Requires a site parameter.

[code language='php']
$data = $this->envato_api->public_threads_active('sitename');
$this->evanto_api->clear();
[/code]

Number of Files in Categories

Shows the number of files in the major categories of a particular site. Requires a site paramater.

[code language='php']
$data = $this->envato_api->public_files_number('sitename');
$this->evanto_api->clear();
[/code]

Category New Files

New files, recently uploaded to a particular site. Requires site and category paramaters.

[code language='php']
$data = $this->envato_api->public_files_new('sitename', 'categoryname');
$this->evanto_api->clear();
[/code]

Popular Files

Returns the popular files for a particular site. Requires a site paramater.

[code language='php']
$data = $this->envato_api->public_files_popular('sitename');
$this->evanto_api->clear();
[/code]

New User Files

Shows the newest 10 files a user has uploaded to a particualr site. Requires username and site paramaters.

[code language='php']
$data = $this->envato_api->public_files_user_new('username', 'sitename');
$this->evanto_api->clear();
[/code]

Random New Files

Shows a random list of newly uploaded files from a particular site (i.e. like the homepage). Requires a site paramater.

[code language='php']
$data = $this->envato_api->public_files_new_random('sitename');
$this->evanto_api->clear();
[/code]

API Releases

Returns Release and Set information for the API (used to generate the documentation).

[code language='php']
$data = $this->envato_api->public_releases();
$this->evanto_api->clear();
[/code]

Private User Functions

These are functions that DO require a username and API key to access. They include things such as account information, billing statements, etc. Make sure to set your username & api key inside the script or with the config function above.

User Vitals

Return a users username and balance.

[code language='php']
$data = $this->envato_api->user_vitals();
$this->evanto_api->clear();
[/code]

User Account

Returns the first name, surname, total earnings, total deposits, balance (deposits + earnings), country and current commission rate for sales.

[code language='php']
$data = $this->envato_api->user_account();
$this->evanto_api->clear();
[/code]

User Earnings & Monthly Sales

Returns the monthly sales data, as displayed on the user’s earnings page.

[code language='php']
$data = $this->envato_api->user_earnings();
$this->evanto_api->clear();
[/code]

User Statement

Returns the last 100 events as seen on the user’s statement page.

[code language='php']
$data = $this->envato_api->user_statement();
$this->evanto_api->clear();
[/code]

User Recent Sales

Shows the 50 recent sales of the user’s items.

[code language='php']
$data = $this->envato_api->user_recent_sales();
$this->evanto_api->clear();
[/code]

User Multiple

This function will return multiple sets of data at once. if you dont supply anything in the parameters it will return all of the results by default. The options inside the array below are the only options you have. You can use as many or as little as you need.

[code language='php']
$data = $this->envato_api->user_multiple(array('account', 'vitals', 'earnings-adn-sales-by-month', 'statement', 'recent-sales'));
$this->evanto_api->clear();
[/code]

Done!

We are all done! If you have any questions, suggestions, bugs or just wanna say thanks then reply in the comments.