XHCI: Completely rewrite Transfer Descriptor and related logic.

This is essentially a complete rewrite of the way TDs work, and a
significant change to how "normal" requests are submitted. In summation:
 * There is now no such thing as a "descriptor chain". This was a concept
   mostly carried over from EHCI and previous controllers that does not
   really map to XHCI properly, as one "transfer descriptor" can contain
   as many TRBs as are necessary to complete the transfer.

   What we were really doing before was setting up multiple TRB sequences
   owned by our xhci_td structure, and then linking them together with the
   CHAIN bit set on each TRB, which in XHCI terms is *one* descriptor,
   not multiple ones as we had it, and we were allocating the same amount
   of memory anyway, so we might as well do it all in one structure.

   So now xhci_td does not have a fixed size of TRBs, but rather a dynamic
   one, and thus also a dynamic number of buffers. As part of this refactor,
   xhci_td is now a data structure for our own use only, and so it is allocated
   from the normal heap instead of as physical memory, and TRBs separately.
 * Removing the distinction between "descriptor" and "descriptor chain" greatly
   simplifies quite a lot of logic related to transfer handling, especially
   in WriteDescriptor and ReadDescriptor, which no longer need to handle
   writing to buffers across xhci_td boundaries.
 * There is now a proper split between "trb_count" and "trb_used". The former
   is the actual number of TRBs allocated in the data structure; the latter
   is the number presently used in said data structure. This clarifies
   quite a lot of code.

As part of this refactor, a number of other related issues were also cleaned up:
 * TRB buffer sizes are now hard-coded to be 4x the Max Packet Size for the
   given pipe. Previously they were 64KB, which is much larger than the spec
   suggests. As we now size them exactly this way, we can set the endpoint's
   Average TRB Length to this value also, which allows the controller to
   better schedule transfers for us. This logic can probably be cleaned up
   further in the future, even.
 * We now write the "TD Size" field of Normal transfers (i.e. packet count,
   not TRB count) properly, as it is now much easier to compute based on
   our new TRB sizing logic.
 * We now record the Max Packet Size for the endpoint in the endpoint
   structure. (This will probably be more useful when we are dealing
   with isochronous transfers later on.)
 * Write the last cycle bit in _LinkDescriptorForPipe after everything else
   has been written. This probably does not affect anything too seriously,
   but it is technically more correct.
 * Added section & page references to the specification in various comments.
 * Added more error checking where applicable.

Some things still to be done that I noticed while working on this change:
 * When we use a single large buffer and manually segment it rather than
   calling AllocateChunk() for each one, there is a massive performance
   gain (I saw 50MB/s -> 95MB/s in testing on some USB devices.) However,
   we can't do this unconditionally, as the stack doesn't allocate physical
   chunks larger than 32 * B_PAGE_SIZE, which is a problem for some filesystems
   which can read/write in chunks of 1MB. I'll implement this in a later commit.
 * Setting the IOC bit on the last TRB in a transfer is acceptable to find
   out when the transfer is finished, but it isn't enough to get a proper
   tally of how much data was transfered. This is what Event Status TRBs
   are for; we should implement them.
 * _LinkDescriptorForPipe has an overflow condition. I'll fix that in the
   next commit.

Tested on a ThinkPad E550 (Intel Broadwell) with usb_disk and usb_hid.

src/add-ons/kernel/busses/usb/xhci.cpp

@@ -1,5 +1,5 @@
 /*
- * Copyright 2011-2019, Haiku Inc. All rights reserved.
+ * Copyright 2011-2019, Haiku, Inc. All rights reserved.
  * Distributed under the terms of the MIT License.
  *
  * Authors:
@@ -703,7 +703,9 @@ XHCI::SubmitControlRequest(Transfer *transfer)
 
 	TRACE("SubmitControlRequest() length %d\n", requestData->Length);
 
-	xhci_td *descriptor = CreateDescriptor(requestData->Length);
+	xhci_td *descriptor = CreateDescriptor(3, requestData->Length);
+	if (descriptor == NULL)
+		return B_NO_MEMORY;
 	descriptor->transfer = transfer;
 
 	// Setup Stage
@@ -723,7 +725,7 @@ XHCI::SubmitControlRequest(Transfer *transfer)
 
 	// Data Stage (if any)
 	if (requestData->Length > 0) {
-		descriptor->trbs[index].qwtrb0 = descriptor->buffer_phy[0];
+		descriptor->trbs[index].qwtrb0 = descriptor->buffer_addrs[0];
 		descriptor->trbs[index].dwtrb2 = TRB_2_IRQ(0)
 			| TRB_2_BYTES(requestData->Length)
 			| TRB_2_TD_SIZE(0);
@@ -734,7 +736,7 @@ XHCI::SubmitControlRequest(Transfer *transfer)
 
 		if (!directionIn) {
 			transfer->PrepareKernelAccess();
-			memcpy(descriptor->buffer_log[0],
+			memcpy(descriptor->buffers[0],
 				(uint8 *)transfer->Vector()[0].iov_base, requestData->Length);
 		}
 
@@ -751,7 +753,7 @@ XHCI::SubmitControlRequest(Transfer *transfer)
 		// Status Stage is an OUT transfer when the device is sending data.
 		// (XHCI 1.1 § 4.11.2.2 Table 4-6 p205.)
 
-	descriptor->trb_count = index + 1;
+	descriptor->trb_used = index + 1;
 
 	status = _LinkDescriptorForPipe(descriptor, endpoint);
 	if (status != B_OK) {
@@ -788,65 +790,60 @@ XHCI::SubmitNormalRequest(Transfer *transfer)
 	if (status != B_OK)
 		return status;
 
-	int32 trbCount = 0;
+	// Compute the size to use for the TRBs, and then how many TRBs
-	xhci_td *descriptor = CreateDescriptorChain(transfer->DataLength(),
+	// of this size we will need. We always need at least 1, of course.
-		&trbCount);
+	const size_t dataLength = transfer->DataLength(),
-	if (descriptor == NULL)
+		maxPacketSize = pipe->MaxPacketSize(),
+		packetsPerTrb = 4;
+	const size_t trbSize = packetsPerTrb * maxPacketSize;
+	int32 trbCount = (dataLength + trbSize - 1) / trbSize;
+
+	xhci_td *td = CreateDescriptor(trbCount, trbSize);
+	if (td == NULL)
 		return B_NO_MEMORY;
 
-	xhci_td *td_chain = descriptor;
-	xhci_td *last = descriptor;
-	int32 rest = trbCount - 1;
-
 	// Normal Stage
-	while (td_chain != NULL) {
+	size_t remaining = dataLength;
-		td_chain->trb_count = td_chain->buffer_count;
+	int32 remainingPackets = (remaining - trbSize) / maxPacketSize;
-		uint8 index;
+	for (int32 i = 0; i < trbCount; i++) {
-		for (index = 0; index < td_chain->buffer_count; index++) {
+		// The "TD Size" field of a transfer TRB indicates the number of
-			td_chain->trbs[index].qwtrb0 = descriptor->buffer_phy[index];
+		// remaining maximum-size *packets* in this TD, *not* including the
-			// TODO: TD Size is actually supposed to be number of packets
+		// packets in the current TRB, and capped at 31 if there are more
-			// remaining, *not* number of TRBs remaining as it is presently.
+		// than 31 packets remaining in the TD. (XHCI 1.1 § 4.11.2.4 p210.)
-			td_chain->trbs[index].dwtrb2 = TRB_2_IRQ(0)
+		int32 tdSize = remainingPackets > 31 ? 31 : remainingPackets;
-				| TRB_2_BYTES(descriptor->buffer_size[index])
+		if (tdSize < 0)
-				| TRB_2_TD_SIZE(rest > 31 ? 31 : rest);
+			tdSize = 0;
-			td_chain->trbs[index].dwtrb3 = B_HOST_TO_LENDIAN_INT32(
+		int32 trbLength = remaining < trbSize ? remaining : trbSize;
-				TRB_3_TYPE(TRB_TYPE_NORMAL) | TRB_3_CYCLE_BIT | TRB_3_CHAIN_BIT
-					| (directionIn ? TRB_3_ISP_BIT : 0));
-			rest--;
-		}
-		// link next td, if any
-		if (td_chain->next_chain != NULL) {
-			td_chain->trbs[td_chain->trb_count].qwtrb0 =
-				td_chain->next_chain->this_phy;
-			td_chain->trbs[td_chain->trb_count].dwtrb2 = TRB_2_IRQ(0);
-			td_chain->trbs[td_chain->trb_count].dwtrb3
-				= B_HOST_TO_LENDIAN_INT32(TRB_3_TYPE(TRB_TYPE_LINK)
-					| TRB_3_CYCLE_BIT | TRB_3_CHAIN_BIT);
-		}
 
-		last = td_chain;
+		td->trbs[i].qwtrb0 = td->buffer_addrs[i];
-		td_chain = td_chain->next_chain;
+		td->trbs[i].dwtrb2 = TRB_2_IRQ(0)
+			| TRB_2_BYTES(trbLength)
+			| TRB_2_TD_SIZE(tdSize);
+		td->trbs[i].dwtrb3 = TRB_3_TYPE(TRB_TYPE_NORMAL)
+			| TRB_3_CYCLE_BIT | TRB_3_CHAIN_BIT
+			| (directionIn ? TRB_3_ISP_BIT : 0);
+
+		td->trb_used++;
+		remaining -= trbLength;
+		remainingPackets -= packetsPerTrb;
 	}
 
-	if (last->trb_count > 0) {
+	// Set the IOC (Interrupt On Completion) bit and unset the CHAIN bit on
-		last->trbs[last->trb_count - 1].dwtrb3
+	// the final TRB in the transfer. (XHCI 1.1 § 6.4.1.1 Table 6-22 p443.)
-			&= B_HOST_TO_LENDIAN_INT32(~TRB_3_CHAIN_BIT);
+	td->trbs[td->trb_used - 1].dwtrb3 &= ~TRB_3_CHAIN_BIT;
-		last->trbs[last->trb_count - 1].dwtrb3
+	td->trbs[td->trb_used - 1].dwtrb3 |= TRB_3_IOC_BIT;
-			|= B_HOST_TO_LENDIAN_INT32(TRB_3_IOC_BIT);
-	}
 
 	if (!directionIn) {
 		TRACE("copying out iov count %ld\n", transfer->VectorCount());
 		transfer->PrepareKernelAccess();
-		WriteDescriptorChain(descriptor, transfer->Vector(),
+		WriteDescriptor(td, transfer->Vector(), transfer->VectorCount());
-			transfer->VectorCount());
 	}
 
 	xhci_endpoint *endpoint = (xhci_endpoint *)pipe->ControllerCookie();
-	descriptor->transfer = transfer;
+	td->transfer = transfer;
-	status = _LinkDescriptorForPipe(descriptor, endpoint);
+	status = _LinkDescriptorForPipe(td, endpoint);
 	if (status != B_OK) {
-		FreeDescriptor(descriptor);
+		FreeDescriptor(td);
 		return status;
 	}
 	TRACE("SubmitNormalRequest() request linked\n");
@@ -898,224 +895,152 @@ XHCI::NotifyPipeChange(Pipe *pipe, usb_change change)
 
 
 xhci_td *
-XHCI::CreateDescriptor(size_t bufferSize)
+XHCI::CreateDescriptor(uint32 trbCount, size_t trbBufferSize)
 {
-	xhci_td *result;
+	xhci_td *result = new xhci_td;
-	phys_addr_t physicalAddress;
+	if (result == NULL) {
-
-	if (fStack->AllocateChunk((void **)&result, &physicalAddress,
-			sizeof(xhci_td)) < B_OK) {
 		TRACE_ERROR("failed to allocate a transfer descriptor\n");
 		return NULL;
 	}
 
-	result->this_phy = physicalAddress;
+	uint32 bufferCount = trbCount;
-	result->buffer_size[0] = bufferSize;
-	result->trb_count = 0;
-	result->buffer_count = 1;
-	result->next = NULL;
-	result->next_chain = NULL;
-	if (bufferSize <= 0) {
-		result->buffer_log[0] = NULL;
-		result->buffer_phy[0] = 0;
-		return result;
-	}
 
-	if (fStack->AllocateChunk(&result->buffer_log[0],
+	// We always allocate 1 more TRB than requested, so that
-			&result->buffer_phy[0], bufferSize) < B_OK) {
+	// _LinkDescriptorForPipe() has room to insert a link TRB.
-		TRACE_ERROR("unable to allocate space for the buffer (size %ld)\n",
+	trbCount++;
-			bufferSize);
+	if (fStack->AllocateChunk((void **)&result->trbs, &result->trb_addr,
-		fStack->FreeChunk(result, result->this_phy, sizeof(xhci_td));
+			(trbCount * sizeof(xhci_trb))) < B_OK) {
+		TRACE_ERROR("failed to allocate TRBs\n");
+		delete result;
 		return NULL;
 	}
+	result->trb_count = trbCount;
+	result->trb_used = 0;
 
-	TRACE("CreateDescriptor allocated buffer_size %ld %p\n",
+	if (trbBufferSize > 0) {
-		result->buffer_size[0], result->buffer_log[0]);
+		// Due to how the USB stack allocates physical memory, we can't just
+		// request one large chunk the size of the transfer, and so instead we
+		// create a series of buffers as requested by our caller.
 
-	return result;
+		// We store the buffer pointers and addresses in one memory block.
-}
+		result->buffers = (void**)calloc(bufferCount,
-
+			(sizeof(void*) + sizeof(phys_addr_t)));
-
+		if (result->buffers == NULL) {
-xhci_td *
+			TRACE_ERROR("unable to allocate space for buffer infos\n");
-XHCI::CreateDescriptorChain(size_t bufferSize, int32 *trbCount)
+			FreeDescriptor(result);
-{
-	size_t packetSize = B_PAGE_SIZE * 16;
-	int32 trbsTotal = (bufferSize + packetSize - 1) / packetSize;
-	if (trbsTotal == 0)
-		trbsTotal = 1;
-	*trbCount = trbsTotal;
-
-	// keep one trb for linking
-	int32 tdCount = (trbsTotal + XHCI_MAX_TRBS_PER_TD - 2)
-		/ (XHCI_MAX_TRBS_PER_TD - 1);
-
-	xhci_td *first = NULL;
-	xhci_td *last = NULL;
-	for (int32 i = 0; i < tdCount; i++) {
-		xhci_td *descriptor = CreateDescriptor(0);
-		if (descriptor == NULL) {
-			FreeDescriptor(first);
 			return NULL;
 		}
-		if (first == NULL)
+		result->buffer_addrs = (phys_addr_t*)&result->buffers[bufferCount];
-			first = descriptor;
 
-		uint8 trbs = min_c(trbsTotal, XHCI_MAX_TRBS_PER_TD - 1);
+		for (uint32 i = 0; i < bufferCount; i++) {
-		TRACE("CreateDescriptorChain trbs %d for td %" B_PRId32 "\n", trbs, i);
+			if (fStack->AllocateChunk(&result->buffers[i],
-		for (int j = 0; j < trbs; j++) {
+					&result->buffer_addrs[i], trbBufferSize) < B_OK) {
-			if (fStack->AllocateChunk(&descriptor->buffer_log[j],
+				TRACE_ERROR("unable to allocate space for the buffer (size %ld)\n",
-					&descriptor->buffer_phy[j],
+					trbBufferSize);
-					min_c(packetSize, bufferSize)) < B_OK) {
+				FreeDescriptor(result);
-				TRACE_ERROR("unable to allocate space for the buffer (size %"
-					B_PRIuSIZE ")\n", bufferSize);
 				return NULL;
 			}
-
-			descriptor->buffer_size[j] = min_c(packetSize, bufferSize);
-			bufferSize -= descriptor->buffer_size[j];
-			TRACE("CreateDescriptorChain allocated %ld for trb %d\n",
-				descriptor->buffer_size[j], j);
 		}
-
+	} else {
-		descriptor->buffer_count = trbs;
+		result->buffers = NULL;
-		trbsTotal -= trbs;
+		result->buffer_addrs = NULL;
-		if (last != NULL)
-			last->next_chain = descriptor;
-		last = descriptor;
 	}
+	result->buffer_size = trbBufferSize;
+	result->buffer_count = bufferCount;
 
-	return first;
+	// Initialize all other fields.
+	result->transfer = NULL;
+	result->trb_completion_code = 0;
+	result->trb_left = 0;
+	result->next = NULL;
+
+	TRACE("CreateDescriptor allocated buffer_size %ld (%p)\n",
+		result->buffer_size, result->buffer);
+
+	return result;
 }
 
 
 void
 XHCI::FreeDescriptor(xhci_td *descriptor)
 {
-	while (descriptor != NULL) {
+	if (descriptor == NULL)
-		for (int i = 0; i < descriptor->buffer_count; i++) {
+		return;
-			if (descriptor->buffer_size[i] == 0)
+
+	if (descriptor->trbs != NULL) {
+		fStack->FreeChunk(descriptor->trbs, descriptor->trb_addr,
+			(descriptor->trb_count * sizeof(xhci_trb)));
+	}
+	if (descriptor->buffers != NULL) {
+		for (uint32 i = 0; i < descriptor->buffer_count; i++) {
+			if (descriptor->buffers[i] == NULL)
 				continue;
-			TRACE("FreeDescriptor buffer %d buffer_size %ld %p\n", i,
+			fStack->FreeChunk(descriptor->buffers[i], descriptor->buffer_addrs[i],
-				descriptor->buffer_size[i], descriptor->buffer_log[i]);
+				descriptor->buffer_size);
-			fStack->FreeChunk(descriptor->buffer_log[i],
-				descriptor->buffer_phy[i], descriptor->buffer_size[i]);
 		}
-
+		free(descriptor->buffers);
-		xhci_td *next = descriptor->next_chain;
-		fStack->FreeChunk(descriptor, descriptor->this_phy,
-			sizeof(xhci_td));
-		descriptor = next;
 	}
+
+	delete descriptor;
 }
 
 
 size_t
-XHCI::WriteDescriptorChain(xhci_td *descriptor, iovec *vector,
+XHCI::WriteDescriptor(xhci_td *descriptor, iovec *vector, size_t vectorCount)
-	size_t vectorCount)
 {
-	xhci_td *current = descriptor;
+	size_t written = 0;
-	uint8 trbIndex = 0;
-	size_t actualLength = 0;
-	uint8 vectorIndex = 0;
-	size_t vectorOffset = 0;
-	size_t bufferOffset = 0;
 
-	while (current != NULL) {
+	size_t bufIdx = 0, bufUsed = 0;
-		if (current->buffer_log[0] == NULL)
+	for (size_t vecIdx = 0; vecIdx < vectorCount; vecIdx++) {
-			break;
+		size_t length = vector[vecIdx].iov_len;
 
-		while (true) {
+		while (length > 0 && bufIdx < descriptor->buffer_count) {
-			size_t length = min_c(current->buffer_size[trbIndex] - bufferOffset,
+			size_t toCopy = min_c(length, descriptor->buffer_size - bufUsed);
-				vector[vectorIndex].iov_len - vectorOffset);
+			memcpy((uint8 *)descriptor->buffers[bufIdx] + bufUsed,
+				(uint8 *)vector[vecIdx].iov_base + (vector[vecIdx].iov_len - length),
+				toCopy);
 
-			TRACE("copying %ld bytes to bufferOffset %ld from"
+			written += toCopy;
-				" vectorOffset %ld at index %d of %ld\n", length, bufferOffset,
+			bufUsed += toCopy;
-				vectorOffset, vectorIndex, vectorCount);
+			length -= toCopy;
-			memcpy((uint8 *)current->buffer_log[trbIndex] + bufferOffset,
+			if (bufUsed == descriptor->buffer_size) {
-				(uint8 *)vector[vectorIndex].iov_base + vectorOffset, length);
+				bufIdx++;
-
+				bufUsed = 0;
-			actualLength += length;
-			vectorOffset += length;
-			bufferOffset += length;
-
-			if (vectorOffset >= vector[vectorIndex].iov_len) {
-				if (++vectorIndex >= vectorCount) {
-					TRACE("wrote descriptor chain (%ld bytes, no more vectors)\n",
-						actualLength);
-					return actualLength;
-				}
-
-				vectorOffset = 0;
-			}
-
-			if (bufferOffset >= current->buffer_size[trbIndex]) {
-				bufferOffset = 0;
-				if (++trbIndex >= current->buffer_count)
-					break;
 			}
 		}
-
-		current = current->next_chain;
-		trbIndex = 0;
 	}
 
-	TRACE("wrote descriptor chain (%ld bytes)\n", actualLength);
+	TRACE("wrote descriptor (%" B_PRIuSIZE " bytes)\n", written);
-	return actualLength;
+	return written;
 }
 
 
 size_t
-XHCI::ReadDescriptorChain(xhci_td *descriptor, iovec *vector,
+XHCI::ReadDescriptor(xhci_td *descriptor, iovec *vector, size_t vectorCount)
-	size_t vectorCount)
 {
-	xhci_td *current = descriptor;
+	size_t read = 0;
-	uint8 trbIndex = 0;
-	size_t actualLength = 0;
-	uint8 vectorIndex = 0;
-	size_t vectorOffset = 0;
-	size_t bufferOffset = 0;
 
-	while (current != NULL) {
+	size_t bufIdx = 0, bufUsed = 0;
-		if (current->buffer_log[0] == NULL)
+	for (size_t vecIdx = 0; vecIdx < vectorCount; vecIdx++) {
-			break;
+		size_t length = vector[vecIdx].iov_len;
 
-		while (true) {
+		while (length > 0 && bufIdx < descriptor->buffer_count) {
-			size_t length = min_c(current->buffer_size[trbIndex] - bufferOffset,
+			size_t toCopy = min_c(length, descriptor->buffer_size - bufUsed);
-				vector[vectorIndex].iov_len - vectorOffset);
+			memcpy((uint8 *)vector[vecIdx].iov_base + (vector[vecIdx].iov_len - length),
+				(uint8 *)descriptor->buffers[bufIdx] + bufUsed, toCopy);
 
-			TRACE("copying %ld bytes to vectorOffset %ld from"
+			read += toCopy;
-				" bufferOffset %ld at index %d of %ld\n", length, vectorOffset,
+			bufUsed += toCopy;
-				bufferOffset, vectorIndex, vectorCount);
+			length -= toCopy;
-			memcpy((uint8 *)vector[vectorIndex].iov_base + vectorOffset,
+			if (bufUsed == descriptor->buffer_size) {
-				(uint8 *)current->buffer_log[trbIndex] + bufferOffset, length);
+				bufIdx++;
-
+				bufUsed = 0;
-			actualLength += length;
-			vectorOffset += length;
-			bufferOffset += length;
-
-			if (vectorOffset >= vector[vectorIndex].iov_len) {
-				if (++vectorIndex >= vectorCount) {
-					TRACE("read descriptor chain (%ld bytes, no more vectors)\n",
-						actualLength);
-					return actualLength;
-				}
-				vectorOffset = 0;
-
-			}
-
-			if (bufferOffset >= current->buffer_size[trbIndex]) {
-				bufferOffset = 0;
-				if (++trbIndex >= current->buffer_count)
-					break;
 			}
 		}
-
-		current = current->next_chain;
-		trbIndex = 0;
 	}
 
-	TRACE("read descriptor chain (%ld bytes)\n", actualLength);
+	TRACE("read descriptor (%" B_PRIuSIZE " bytes)\n", read);
-	return actualLength;
+	return read;
 }
 
 
@@ -1284,13 +1209,14 @@ XHCI::AllocateDevice(Hub *parent, int8 hubAddress, uint8 hubPort,
 		return NULL;
 	}
 
+	mutex_init(&device->endpoints[0].lock, "xhci endpoint lock");
 	device->endpoints[0].device = device;
+	device->endpoints[0].max_packet_size = maxPacketSize;
 	device->endpoints[0].td_head = NULL;
-	device->endpoints[0].trbs = device->trbs;
 	device->endpoints[0].used = 0;
 	device->endpoints[0].current = 0;
+	device->endpoints[0].trbs = device->trbs;
 	device->endpoints[0].trb_addr = device->trb_addr;
-	mutex_init(&device->endpoints[0].lock, "xhci endpoint lock");
 
 	// device should get to addressed state (bsr = 0)
 	if (SetAddress(device->input_ctx_addr, false, slot) != B_OK) {
@@ -1365,6 +1291,8 @@ XHCI::AllocateDevice(Hub *parent, int8 hubAddress, uint8 hubPort,
 		_WriteContext(&device->input_ctx->input.dropFlags, 0);
 		_WriteContext(&device->input_ctx->input.addFlags, (1 << 1));
 		EvaluateContext(device->input_ctx_addr, device->slot);
+
+		device->endpoints[0].max_packet_size = deviceDescriptor.max_packet_size_0;
 	}
 
 	Device *deviceObject = NULL;
@@ -1490,21 +1418,22 @@ XHCI::_InsertEndpointForPipe(Pipe *pipe)
 			EvaluateContext(device->input_ctx_addr, device->slot);
 		}
 
+		mutex_init(&device->endpoints[id].lock, "xhci endpoint lock");
+		MutexLocker endpointLocker(device->endpoints[id].lock);
+
 		device->endpoints[id].device = device;
-		device->endpoints[id].trbs = device->trbs
+		device->endpoints[id].max_packet_size = pipe->MaxPacketSize();
-			+ id * XHCI_MAX_TRANSFERS;
-		device->endpoints[id].trb_addr = device->trb_addr
-			+ id * XHCI_MAX_TRANSFERS * sizeof(xhci_trb);
 		device->endpoints[id].td_head = NULL;
 		device->endpoints[id].used = 0;
 		device->endpoints[id].current = 0;
 
+		device->endpoints[id].trbs = device->trbs
+			+ id * XHCI_MAX_TRANSFERS;
+		device->endpoints[id].trb_addr = device->trb_addr
+			+ id * XHCI_MAX_TRANSFERS * sizeof(xhci_trb);
 		memset(device->endpoints[id].trbs, 0,
 			sizeof(xhci_trb) * XHCI_MAX_TRANSFERS);
 
-		mutex_init(&device->endpoints[id].lock, "xhci endpoint lock");
-		MutexLocker endpointLocker(device->endpoints[id].lock);
-
 		TRACE("_InsertEndpointForPipe trbs device %p endpoint %p\n",
 			device->trbs, device->endpoints[id].trbs);
 		TRACE("_InsertEndpointForPipe trb_addr device 0x%" B_PRIxPHYSADDR
@@ -1591,10 +1520,7 @@ XHCI::_LinkDescriptorForPipe(xhci_td *descriptor, xhci_endpoint *endpoint)
 	}
 
 	endpoint->used++;
-	if (endpoint->td_head == NULL)
+	descriptor->next = endpoint->td_head;
-		descriptor->next = NULL;
-	else
-		descriptor->next = endpoint->td_head;
 	endpoint->td_head = descriptor;
 
 	uint8 current = endpoint->current;
@@ -1602,34 +1528,32 @@ XHCI::_LinkDescriptorForPipe(xhci_td *descriptor, xhci_endpoint *endpoint)
 
 	TRACE("_LinkDescriptorForPipe current %d, next %d\n", current, next);
 
-	xhci_td *last = descriptor;
+	// Compute next link
-	while (last->next_chain != NULL)
+	addr_t addr = endpoint->trb_addr + next * sizeof(xhci_trb);
-		last = last->next_chain;
+	descriptor->trbs[descriptor->trb_used].qwtrb0 = addr;
-
+	descriptor->trbs[descriptor->trb_used].dwtrb2 = TRB_2_IRQ(0);
-	// compute next link
+	descriptor->trbs[descriptor->trb_used].dwtrb3 = B_HOST_TO_LENDIAN_INT32(
-	addr_t addr = endpoint->trb_addr + next * sizeof(struct xhci_trb);
-	last->trbs[last->trb_count].qwtrb0 = addr;
-	last->trbs[last->trb_count].dwtrb2 = TRB_2_IRQ(0);
-	last->trbs[last->trb_count].dwtrb3 = B_HOST_TO_LENDIAN_INT32(
 		TRB_3_TYPE(TRB_TYPE_LINK) | TRB_3_CYCLE_BIT);
 
 	endpoint->trbs[next].qwtrb0 = 0;
 	endpoint->trbs[next].dwtrb2 = 0;
 	endpoint->trbs[next].dwtrb3 = 0;
 
-	// link the descriptor
+	// Link the descriptor.
-	endpoint->trbs[current].qwtrb0 = descriptor->this_phy;
+	endpoint->trbs[current].qwtrb0 = descriptor->trb_addr;
 	endpoint->trbs[current].dwtrb2 = TRB_2_IRQ(0);
-	endpoint->trbs[current].dwtrb3 = B_HOST_TO_LENDIAN_INT32(
+	endpoint->trbs[current].dwtrb3 = TRB_3_TYPE(TRB_TYPE_LINK);
-		TRB_3_TYPE(TRB_TYPE_LINK) | TRB_3_CYCLE_BIT);
+
+	// Everything is ready, so write the cycle bit.
+	endpoint->trbs[current].dwtrb3 |= TRB_3_CYCLE_BIT;
 
 	TRACE("_LinkDescriptorForPipe pCurrent %p phys 0x%" B_PRIxPHYSADDR
 		" 0x%" B_PRIxPHYSADDR " 0x%08" B_PRIx32 "\n", &endpoint->trbs[current],
 		endpoint->trb_addr + current * sizeof(struct xhci_trb),
 		endpoint->trbs[current].qwtrb0,
 		B_LENDIAN_TO_HOST_INT32(endpoint->trbs[current].dwtrb3));
-	endpoint->current = next;
 
+	endpoint->current = next;
 	return B_OK;
 }
 
@@ -1723,12 +1647,12 @@ XHCI::ConfigureEndpoint(uint8 slot, uint8 number, uint8 type, uint64 ringAddr,
 		case 3:
 		case 5:
 		case 7:
-			dwendpoint4 = ENDPOINT_4_AVGTRBLENGTH(min_c(maxFrameSize,
+			dwendpoint4 = ENDPOINT_4_AVGTRBLENGTH(maxPacketSize * 4)
-				B_PAGE_SIZE)) | ENDPOINT_4_MAXESITPAYLOAD((
+				| ENDPOINT_4_MAXESITPAYLOAD((
 					(maxBurst+1) * maxPacketSize));
 			break;
 		default:
-			dwendpoint4 = ENDPOINT_4_AVGTRBLENGTH(B_PAGE_SIZE);
+			dwendpoint4 = ENDPOINT_4_AVGTRBLENGTH(maxPacketSize * 4);
 			break;
 	}
 
@@ -2112,37 +2036,34 @@ XHCI::HandleTransferComplete(xhci_trb* trb)
 	uint32 remainder = TRB_2_REM_GET(trb->dwtrb2);
 
 	for (xhci_td *td = endpoint->td_head; td != NULL; td = td->next) {
-		for (xhci_td *td_chain = td; td_chain != NULL;
+		int64 offset = (source - td->trb_addr) / sizeof(xhci_trb);
-				td_chain = td_chain->next_chain) {
+		if (offset < 0 || offset >= td->trb_count)
-			int64 offset = (source - td_chain->this_phy) / sizeof(xhci_trb);
+			continue;
-			if (offset < 0 || offset >= XHCI_MAX_TRBS_PER_TD)
-				continue;
 
-			TRACE("HandleTransferComplete td %p trb %" B_PRId64 " found\n",
+		TRACE("HandleTransferComplete td %p trb %" B_PRId64 " found\n",
-				td_chain, offset);
+			td, offset);
 
-			// The TRB at offset trb_count will be the link TRB, which we do not
+		// The TRB at offset trb_used will be the link TRB, which we do not
-			// care about (and should not generate an interrupt at all.)
+		// care about (and should not generate an interrupt at all.)
-			// We really care about the properly last TRB, at index "count - 1".
+		// We really care about the properly last TRB, at index "count - 1".
-			if (offset == td_chain->trb_count - 1) {
+		if (offset == td->trb_used - 1) {
-				_UnlinkDescriptorForPipe(td, endpoint);
+			_UnlinkDescriptorForPipe(td, endpoint);
-				endpointLocker.Unlock();
+			endpointLocker.Unlock();
 
-				td->trb_completion_code = completionCode;
+			td->trb_completion_code = completionCode;
-				td->trb_left = remainder;
+			td->trb_left = remainder;
-				// add descriptor to finished list
+			// add descriptor to finished list
-				Lock();
+			Lock();
-				td->next = fFinishedHead;
+			td->next = fFinishedHead;
-				fFinishedHead = td;
+			fFinishedHead = td;
-				Unlock();
+			Unlock();
-				release_sem(fFinishTransfersSem);
+			release_sem(fFinishTransfersSem);
-				TRACE("HandleTransferComplete td %p done\n", td);
+			TRACE("HandleTransferComplete td %p done\n", td);
-			} else {
+		} else {
-				TRACE_ERROR("TRB %" B_PRIxADDR " was found, but it wasn't the "
+			TRACE_ERROR("TRB %" B_PRIxADDR " was found, but it wasn't the "
-					"last in the TD!\n", source);
+				"last in the TD!\n", source);
-			}
-			return;
 		}
+		return;
 	}
 	TRACE_ERROR("TRB 0x%" B_PRIxADDR " was not found in the endpoint!\n", source);
 }
@@ -2500,11 +2421,11 @@ XHCI::FinishTransfers()
 						TRACE("copying in data %d bytes\n", requestData->Length);
 						transfer->PrepareKernelAccess();
 						memcpy((uint8 *)transfer->Vector()[0].iov_base,
-							td->buffer_log[0], requestData->Length);
+							td->buffers[0], requestData->Length);
 					} else {
 						TRACE("copying in iov count %ld\n", transfer->VectorCount());
 						transfer->PrepareKernelAccess();
-						ReadDescriptorChain(td, transfer->Vector(),
+						ReadDescriptor(td, transfer->Vector(),
 							transfer->VectorCount());
 					}
 				}

src/add-ons/kernel/busses/usb/xhci.h

@@ -34,31 +34,36 @@ enum xhci_state {
 
 
 typedef struct xhci_td {
-	struct xhci_trb	trbs[XHCI_MAX_TRBS_PER_TD];
+	xhci_trb*	trbs;
+	phys_addr_t	trb_addr;
+	uint32		trb_count;
+	uint32		trb_used;
 
-	phys_addr_t	this_phy;				// A physical pointer to this address
+	void**		buffers;
-	phys_addr_t	buffer_phy[XHCI_MAX_TRBS_PER_TD];
+	phys_addr_t* buffer_addrs;
-	void	*buffer_log[XHCI_MAX_TRBS_PER_TD];	// Pointer to the logical buffer
+	size_t		buffer_size;
-	size_t	buffer_size[XHCI_MAX_TRBS_PER_TD];	// Size of the buffer
+	uint32		buffer_count;
-	uint8	buffer_count;
 
-	struct xhci_td	*next_chain;
+	Transfer*	transfer;
-	struct xhci_td	*next;
+	uint8		trb_completion_code;
-	Transfer *transfer;
+	uint32		trb_left;
-	uint8	trb_count;
+
-	uint8	trb_completion_code;
+	xhci_td*	next;
-	uint32	trb_left;
+} xhci_td;
-} xhci_td __attribute__((__aligned__(16)));
 
 
 typedef struct xhci_endpoint {
-	xhci_device		*device;
+	mutex 			lock;
-	xhci_td 		*td_head;
+
-	struct xhci_trb *trbs; // [XHCI_MAX_TRANSFERS]
+	xhci_device*	device;
-	phys_addr_t trb_addr;
+	size_t			max_packet_size;
-	uint8	used;
+
-	uint8	current;
+	xhci_td*		td_head;
-	mutex	lock;
+	uint8			used;
+	uint8			current;
+
+	xhci_trb*		trbs; // [XHCI_MAX_TRANSFERS]
+	phys_addr_t 	trb_addr;
 } xhci_endpoint;
 
 
@@ -139,15 +144,14 @@ private:
 	static	int32				FinishThread(void *data);
 			void				FinishTransfers();
 
-			// Descriptor
+			// Descriptor management
-			xhci_td *			CreateDescriptor(size_t bufferSize);
+			xhci_td *			CreateDescriptor(uint32 trbCount,
-			xhci_td *			CreateDescriptorChain(size_t bufferSize,
+									size_t trbBufferSize);
-									int32 &trbCount);
 			void				FreeDescriptor(xhci_td *descriptor);
 
-			size_t				WriteDescriptorChain(xhci_td *descriptor,
+			size_t				WriteDescriptor(xhci_td *descriptor,
 									iovec *vector, size_t vectorCount);
-			size_t				ReadDescriptorChain(xhci_td *descriptor,
+			size_t				ReadDescriptor(xhci_td *descriptor,
 									iovec *vector, size_t vectorCount);
 
 			status_t			_LinkDescriptorForPipe(xhci_td *descriptor,

src/add-ons/kernel/busses/usb/xhci_hardware.h

@@ -309,7 +309,6 @@
 #define XHCI_MAX_SCRATCHPADS	256
 #define XHCI_MAX_DEVICES		128
 #define XHCI_MAX_TRANSFERS		8
-#define XHCI_MAX_TRBS_PER_TD	18
 
 
 struct xhci_trb {