SCSI I/O dispatch and return path

As discussed in previous blog scsi_request_fn is the final function called for SCSI devices.

SCSI DISPATCH to HBA

scsi_request_fn takes out the scsi_device from q->queuedata(This queuedata sdev is filled in scsi_alloc_sdev()). Also it takes out the scsi_host from the scsi_device. Now it starts a infinite for loop to take out the requests from request_queue. The function to take out requests is blk_peek_request().

blk_peek_request forms the scsi_cmnd and fills it in the req->special field(scsi_prep_fn). Now from the request it takes out the scsi_cmnd (from req->special).

scsi_prep_fn calls
scsi_setup_blk_pc_cmnd calls
scsi_get_cmd_from_req -- allocates memory for scsi command
scsi_init_io ---- fills the scatter gather table from request to scsi cmnd -> sdb(scsi_data_buffer). This is done in function scsi_init_sgtable
scsi_init_sgtable calls 
blk_rq_map_sg  calls
__blk_bios_map_sg -- walk the list, and fill in the addresses and sizes of each segment for_each_bio this function and for each segment in this bio __blk_segment_map_sg is called.
sg_set_page sets the page, offset and bytes to write

The integrity or checksum data buffers are mapped using in blk_rq_map_integrity_sg function.

scsi_cmnd has the struct scsi_data_buffer sdb; field which has the sg_table with all the pages link, their respective offset and length :

struct scatterlist {
unsigned long page_link;
unsigned int offset;
unsigned int length;
dma_addr_t dma_address;
};
dma_address is filled by the function dma_map_sg.
https://lwn.net/Articles/234617/

This command is sent to scsi_dispatch_cmd function. This command is then sent to HBA's queuecommand function : rtn = host->hostt->queuecommand(host, cmd);

For the Qlogic HBA the function is qla2xxx_queuecommand() (present in scsi_host_template for Qlogic)

qla2xxx_queuecommand: 

from the scsi_cmnd fc_port fc_rport are taken. Now the fc_rport status is checked using fc_remote_port_chkready. After all the checks for the remote port the scsi request block(SRB) is formed. The call back function qla2x00_sp_compl is registered for this scsi request block.

Now the start_scsi(sp) function is called with this SRB. qla24xx_start_scsi is the function registered by Qlogic for sending scsi command to ISP(FC to PCI card).

qla24xx_start_scsi:
HBA has a request and response queue for SCSI requests and responses. It calls dma_map_sg to determine the proper bus address to give to the device.
intel_map_sg is the function which does the mapping on intel platforms. It will call domain_sg_mapping will call __domain_mapping . It fills the dma_address :

sg->dma_address = ((dma_addr_t)iov_pfn << VTD_PAGE_SHIFT) + sg->offset;
iov_pfn is filled using intel IOMMU functions.
READ : DMA-API-HOWTO.txt in linux Documentation directory
Function qla24xx_build_scsi_iocbs gets the bus address of each of the SG entries and their lengths.

DMA transfers on PCI Express bus : 
https://stackoverflow.com/questions/27470885/how-does-dma-work-with-pci-express-devices
Cache snooping and DMA
https://stackoverflow.com/questions/7132284/dma-cache-coherence-management

Interrupt registration by QLA driver
qla2x00_probe_one is called while qlogic driver is loaded. qla2x00_request_irqs function is called to register the irqs.
ret = request_irq(ha->pdev->irq, ha->isp_ops->intr_handler,
   ha->flags.msi_enabled ? 0 : IRQF_SHARED,
   QLA2XXX_DRIVER_NAME, rsp);


RETURN FROM HBA

The HBA will receive an interrupt about the scsi command completion. In case of qlogic hba, the function qla2100_intr_handler() will handle the interrupt. This interrupt handler was configured in the qla2x00_iospace_config() function. This will call the qla2x00_process_response_queue() function. In the qla2x00_process_response_queue() function, the function qla2x00_process_completed_request() is called which calls the qla2x00_sp_compl function.
qla2x00_sp_free_dma is called to free the dma_map (dma_unmap_sg) done while sending the request.
In the qla2x00_sp_compl, the scsi_done function is called.

The scsi_done function will call in turn call __blk_complete_request(). This function will raise the block interrupt which will be handled by the blk_done_softirq() function. This function will call the scsi_softirq_done(). The scsi_softirq_done() function address is added to the request queue object when the scsi queue is allocated for a scsi device in the scsi_alloc_queue(). In case of success, the scsi_finish_command() will be called.

Following is the onward sequence of function calls :

scsi_finish_command -> 

scsi_io_completion -> in case of scsi command request, it calls -> 

blk_end_request_all() -> 

blk_end_bidi_request() -> 

blk_update_bidi_request() -> 

blk_update_request() -> for each bio in the request, -> 

req_bio_endio() ->

bio_endio() which in turn calls the call back function "bi_end_io" registered when the bio was initialized().


Say if mpage_end_io was the function registered then bi_end_io will point to same. mpage_end_io will iterate through all the pages of bio using bio_for_each_segment_all and then call SetPageUptodate. This will unblock the parent page.